Veit finds it hard to talk about the work that he is doing at feenk, working on Glamorous Toolkit, in the “dead” [⇒ Stop Drawing Dead Fish] medium of a blog post. post 
> Consequently, I haven’t written much about anything I’ve done in a long, long while.
That doesn’t mean I (Veit Heller) haven’t done stuff, however! To prove it—and to bridge the gap until I can figure out how to write about things in a more “live” fashion that better suits my current work—here’s a recording of a talk I gave on a Carp IDE I implemented in a few weekends in Glamorous Toolkit. It’s from last year, but the recording only became available two weeks ago.
YOUTUBE G-0Vic9fQU0 Talk presented by Veit Heller at the ESUG 2022 conference.
Create a new work in the Living Library
Create a new work in the Living Library
LlYouTubeVideo fromString: 'https://youtu.be/G-0Vic9fQU0'
Bind it to the variable copy in
self libraryAddCopy: copy
self transcript
self lepiterPage
QuTranscript Page: "Implementing an IDE in GT" by Veit Heller
and export it as HTML:
page := FedWikiPage new url: (thisSnippet previousSnippetDo: [ :s | (s ast // LeExternalLabelUrlLinkNode name) anyOne url ] ifNone: [ ]) asUrl
Hi I'm Veit and today I want to talk about implementing an IDE in GT, the Glamorous Toolkit. But before we go through the how's and what's and the ins and outs of this talk I have to answer the question Why?
I just think IDEs done right are downright delightful and I think in this audience most people would agree. Smalltalkers like their IDEs. But sadly I think for the wider tech community that's not always true. Because most of us are stuck with glorified text editors. We have Ides they're called Ides but really they're a big pain for text and then not much else maybe a text-based debugger but that also doesn't get us very far that's because we don't spend the time and the resources and because for some environments it's just downright hard to make something delightful. And so we resort to our actual text editors because they integrate better with the system some of us might use a text editor that's based in a terminal like Vim or an operating system that also happens to have a text editor like emacs or maybe even resort to notepad, right?
So we end up reading text and writing text and thinking in text, but of course most of you know that code isn't actually text, it happens to be encoded in text, but that's kind of it's incidental, almost um, code doesn't have to be text, code is not prose and so I think we can do better and I think you'd think that too
and so this is my little idea for how something could be built that could be better um this is a module coder for carp it is very tall and skinny in this slide but it doesn't have to be um in it we have the module example which uses the test module in test.carp if you can't read any carp or you don't know carp which is all of you probably don't worry about it we're not even looking at code yet
we have some documentation it doesn't tell us much it says example module documentation but we have documentation and if we click on the methods tab then we actually see what kind of methods are defined in the module
it's two of them one is called X and it's the dynamic which means that it's available at macro expansion time yes carp has macros we're going to talk about that later and the other one has a scary function type that says it takes an integer and a type A that's a type variable and it returns that same type A
that doesn't tell you anything because you're used to Dynamic typing don't worry about it we can expand this just for fun and for the first time we realize we're dealing with a lisp
We haven't looked at any syntax yet but we already understand the module I think that's kind of cool but now we look at syntax and we say see that we're defining a function that takes two arguments A and B uh adds those and before it adds them it will also convert a from an integer from int is what is called an interface in a carp it just means that it's a function with multiple implementations depending on the type so you can convert many things to an in from an integer to its representation now
that's kind of neat um but if we put this in a pager in GT we can actually use the buttons I mean we can use the buttons otherwise too but it's going to spawn somewhere else in the pager if I'm gonna click on this little thing that's going to tell me generate source I now get the text-based source
and if you can't read that because it's too small don't worry about it you don't have to read it but basically it's going to generate you know the documentation um annotation it's going to say you know the final module example that uses test and then inside online four in this tiny module we actually start looking at the code that we care about which is the functions and because I didn't want to deal with that sort of stuff I built a little module coder but that's only half of what we have um this is a lepidor snippet also a regular coder but it's also an integration lepitor so you can just if you load GT for carp which is this integration um and it's available as a regular old uh Baseline if you load that then you can have uh custom Snippets for cart and in this case uh we're defining a function main we're printing something with the i o module and then we're doing something else it doesn't really matter um but more importantly we have the buttons down here and again I'm going to put this in a pager and then we can play around with the buttons but maybe we should spend a second trying to think about what this will result in so it's a case expression and it's going to add zero and one and if it's one it's going to return okay to println and if it's two which it shouldn't be it's not okay at all which is appropriate and if it's something else we don't even know what's happening anymore um if you don't understand the Ampersand and add signals don't worry about them we're gonna gloss over them now and we're gonna mostly gloss over them later now I put this in a pager again and I can use the buttons so the first one will evaluate the second one will inspect inspection gives us nothing because it's just a function definition and a function definition doesn't do anything on its own but because it's the main function we can build an executable with it um a main function is in the old tradition of C and C like languages so we can we can build a main function and build and run and then if we do that then we're going to see the console output okay we're going to see the exit code of the executable which is zero and we can also look at Raw output and so on but that's not so interesting what's more interesting are the other Integrations like expanding the macros in here and then we see that the Ampersand actually gets expanded to ref if you can't read any of that again don't worry it's all lisp it's all a weird language that fell out of time um the at gets expanded to copy and the case is actually a macro that D sugars to let an anesthetif statement all right but what if something goes wrong so in this case we're saying if something is true and then we say true but we're missing an else branch case statements have to have an else branch so if I try to evaluate that I get a debugger but the debugger is on carp code so here it says macro error case has even number of branches at an else Branch that's helpful enough but if we don't understand it we can still step through the execution of the carp code if we want to we have the stack Trace you can look at things and that's kind of helpful it's especially helpful because I write a lot of macros and macros are annoying to debug especially if all you have is what we have on the right here the stack Trace now with the debugger it's a little bit nicer so let's take a step back before we go through how to implement something like that and talk about what Carp Is and I'm going to throw some some lingo at you it's a statically compiled lisp e programming language it's not a list um it just looks like one it has it has macros we've already seen those it's the classic um it's the classic common list death macro kind of thing it has Type inference um in the vein of like Haskell or camel for those type system aficionados amongst yous that's hindley Milner type system or Hindi Milner Damas it's basically uh just building a bunch of unifications and it's it's quite it's quite nice you don't need to worry about types most of the time basically but they're still there it is more closely related to a camel in this vein than Haskell because it doesn't have a lot of the fancy things that Haskell has it does have some in product types which are structs and option types where you have multiple multiple Constructors for a type if none of this means anything to you don't worry I just want to make sure that we're all kind of that we have the background that we need and we have borrow checking as in Rust that is a somewhat simple method of not having uh to garbage collect but also don't have to to manage memory manually so we get to kind of infer lifetimes of things and uh destructors get inserted into the code ahead of time statically it's quite nice it disallows a whole class of pro programs but it also disallows a whole class of bugs and that's a nice trade-off usually um but I mostly saw it as the perfect challenge because it's so far away from small talk I mean it has types it has macros it has all of these things um it helps that I sorry yeah yeah yeah yeah yeah yeah yeah it I'm sorry of course it has static types um and it of course it helps that I'm contributing to the language but mostly I just wanted to see whether I could make it work because if I can make that work with all of those weird features I feel like I can make anything work right after that something like python or JavaScript feels like a breeze I also wrote a little booklet about how to do this um it's part of the GT4 card repository so if you download that you get a little leopard booklet that walks you through the process of building everything about this uh it's a little terse just because there's a lot of moving parts and I didn't want to you know hand you everything you still have to read a little bit of documentation and so on but it gives you all the pointers um as a brief aside this booklet has in the appendix a page called presentation at isoc and where actually if I minimize this inside that page so this is the booklet running inside the presentation running inside the booklet I think that's pretty cool all right so what do we need to make this work this is the technical part of the talk we need a smack parser we need a syntax highlighter we need a language link integration yesterday when we talked about the remote Runner we heard about feral link which is the Pharaoh version of language link language link is the underlying technology that powers these these servers um we need a custom snippet type and with it a custom coder but I'm going to get to that we need a handful of weekends or maybe longer or maybe shorter depending on uh how fast you get started and then views the taste I personally added the module coder that's not applicable to all languages right not all languages have modules not all languages modules look like modules and cards so you will have to think about what you want to your views to look like let's go through the components so first of all we have a parser and if we look at this parser we don't need this sidebar I don't think if we look at this parser it is the it's a standard smack parser it has a bunch of a bunch of rules it's actually quite easy to parse um to parse lisp languages um but it's still kind of involved and that's mostly because I wanted to have a real abstract syntax tree and not a parse tree smack can provide you with both if you don't tell smack what to parse things into it will generate a parse tree and then you have a bunch of a tree with a bunch of text in the nodes but I already told you that I don't think text is a very good medium so I would suggest if you're building your IDE build a real AST and you do that by saying in the source of the parser what you would like your uh Expressions to look like with the prefix and suffix and root and inside all of the parses you say what to parse into this little thing here just means that you should parse this into a node called map the rule is called map and we parse into map and then if we try this out we can test this and I'm just going to use a map I'm going to write some nonsensical code but just to show you maybe uh what we're doing here um so you get a start node which is a list of expressions and then inside the Expressions you get the list node which is this thing so if I click on that I see that's what that parse is and then inside that I have Expressions again a variable node plus a number node one another variable node a and then the map node right so that's kind of how how we parse things and it kind of helps us to have a little bit of structure before we start with the more interesting bits now once we've done that we want to highlight our AST and all you have to do in GT to make that work is you have to add one method to the class GT smack parser Styler in this case it's called a carp Styler takes a parser class we don't care about what that means it will it will have in The annotation the name of the parser that it works with and then it has a bunch of rules and if I expand this again you can see the rules it's just really just you know a keyword should be bold and should be purple and that kind of stuff um I also went the extra Mile and if you see here in in the small talk code kind of the um parentheses have colors and they're color coded according to their depth and if you click on you know the parentheses you see where they start and where they end I wanted to have that for carp too so I added that in the carp Styler utilities if you want something like that for your language you can just take that verbatim and it will work for your language as well they're part of if I remember correctly I'm not um the expert here right there sorry oh yes of course uh whether uh the styler is part of smack or a GT extension if I remember correctly they're part of GT4 smack so the GT extension on top of smack all right then we need a language link and I'm just going to talk about the client here because the client is inside GT the server can be anything that understands the protocol um I personally didn't want to understand the protocol but I wanted a server so I just took the python server and stuck a card process in there and it kind of worked so I still don't understand how the uh how the protocol works but I also don't need to there is an implementation for it if you do want to understand it you're a better person than I am now on the client side we need to tell the server inclined what points to run on because they work in sockets and then we need to say which which message broker strategy in this case HTTP I think we also have message pack if you're interested in that um and then you kind of give it a process you give it a platform a command a command Factory you give it a serializer and the parser that's a lot of stuff but the important bits maybe are the python process the the server process which in this case is the python process it just gives you you know this is this is where the executable is and so on um the platform which basically instantiates a new platform every time you start an application it will make a fresh directory and run carb inside that fresh directory and so on um in the case of python it actually will create a virtual environment for uh your system every time carp doesn't have that notion so we don't do that but you could if your language wants that and then the command factor is not super important I don't think that I did anything there but the command is also interesting because it takes a stream of instructions and it makes code that the server understands from it in this case it just joins all the instructions with space and replaces all Carriage returns by line feeds simple enough but you might want or need something else like I don't know add semicolons or add whatever you want right so that's kind of it that's all you need to do on the uh language link side it's a lot but it's doable and then we just have the snippet the snippet is easy it itself doesn't have a lot what's not easy is what the snippet contains which is the coding the coder is the actual thing that has all the buttons and does all the stuff and executes things but I suggest if you want to start with this you start with the snippet because the snippet holds on to everything else if you start with the coder you might never realize that you even need a snippet now you go with the coder and it has a bunch of stuff but the most important thing is bind and execute find and execute literally will bind all variables that we need do all this stuff and then send and then send whatever execution we need over to the server so that's kind of fits uh it's a mess but it works and now we can take a quick breather because that was all the code that was kind of a lot you do need a lot of code to make an IDE work but really uh I don't think it was too much code it was enough code for me to write it alone on a couple of weekends and we built an IDE a that's pretty doable you know so what I want you to take away even if you don't believe that it's all that doable or if you're not if you don't believe that it's all that great um I think one of the takeaways is that a non-trivial amount of work goes into building an ID well the um but also most of it happens not inside the code but when you think about what might be interesting to add what might be interesting to abstract away and that kind of things I spent more time thinking about how to make modules work in a way that allows me to do everything I do in text but not have all the bloat of text you know so I think that's where more of the work happens but it's also more flexible than sitting down and writing code and I want you to take away that it can be by it can be done by one person um and I think that's pretty cool and lastly I want you to keep sending the message that this this here this IDE it's not the same as this which is a bunch of text and you already know that but and I have to add a little low note to this um before we pick up and it's kind of my fault um no one in the carp Community cared no one wanted that tool because no one understood why it was useful we all have text editors why do we need an IDE and maybe I just didn't they didn't see what I saw and I wasn't able to tell them what I saw in a way that was understandable I think um but some of you might be able to on another note also to make it a little more positive before I leave you is that it doesn't matter it doesn't really matter that no one else cares about this tool because it already made my life writing carp so much better and I've done things that I would otherwise not have done I've written more complex macros because I can debug them more easily that sort of stuff and really then in the end those few hours that I spent we're well spent I think and maybe they will be well spent for you as well if you want you can always contact me if you want to do something like this and you're interested in getting started otherwise have a great day and if you have any questions shoot them at me [Applause] yeah yeah maybe foreign yes it is a specific so the question was whether it's a specific GT based API and yes it is something like language server or something yeah I asked myself the same question I don't have an answer yet I I really I literally I was not I I don't know but maybe Alistair do you want to pipe up sorry um why do we not use something more popular like LSP for like the uh for the server implementation instead of like a custom Language Link thing um so all right okay anything else all right [Applause]