Clustering several emacs instances

Why should we implement emacsen talking to each other? -- Because We Can(TM) :)

Before I go public with my intentions on this topic I prefer doing some research in advance. I have hacked on several different approaches for this and am currently fiddling with the advantages and disadvantages of each of them.

Afaics my nomenclature is slightly misleading: I don't want to cluster emacsen to do parallelized jobs, rather I want them to talk to each other and make them ship not only code that is evaluated at the respective remote side but also pump compiled lisp and stuff forth and back.

Thus we should ponder

• a suited opaque layer to envelope stuff
• a suited method on how to ship stuff

Here are my approaches (in the order of preference):

0. communication via the emacs built-in server listening socket and open-network-stream on the client side

This approach is as easy as we (emacsers) are used to. To establish a tcp server socket listening on port 5700 just eval:

(make-network-server-socket "myserver" "server-buffer" 'tcp '5700)

Other emacsen may connect then via

(open-network-stream "myclient" "client-buffer" some.host.org '5700)

And now we just need to implement the envelope layer to pump compiled lisp and stuff through the pipes. :)

OKAY, AND NOW SEND ME THE PATCH TO MAKE EMACS ACTING AS SERVER. :P

1. communication via ircd

For that approach one would have to

• compile an ircd
• set it up correctly and
• have it run publically

This seems too complicated for general usage. Furthermore (dunno whether good or bad) one box is the central node.

Another point (dunno again whether good or bad) is that others may (mis)use this socket to start a chatting ruckus.

This approach is nonetheless the most promising one. Vast parts are even implemented (see erbot instances on OPN).

2. communication via netcat listening sockets

For that approach one would have to start a netcat instance in listening mode. Not only this, every pair of intercommunicating emacsen need to have one network stream to the (local) netcat process and another to the remote emacs respectively (which in turn is a listening netcat).

That's exactly why this approach is impractical for large scale use.

Sorta discussion I had

It's nice to see that I'm not alone with my urge for clustering emacsen (or rather call them interoperating emacsen).

See here for the discussion I had on #xemacs.

0619.19:18:16 <hroptatyr> 0619.10:54:30 <myrkraverk> do you know how his clustered emacsen project goes?
0619.19:18:33 <hroptatyr> hm, yeah, it's sorta idea now ... i have different approaches that work already
0619.19:18:53 <hroptatyr> but actually all of them depend on some third party resources :\
0619.19:18:54 <myrkraverk> hmm, k
0619.19:18:58 <myrkraverk> oh
0619.19:19:31 <myrkraverk> I've been thinking of some sort of find-remote-file
0619.19:19:36 <hroptatyr> but im not that much into C to provide server sockets nativele
0619.19:19:44 <hroptatyr> hm, for that there's tramp
0619.19:19:47 <myrkraverk> that connects to a running emacs, and uses the current one only as a display engine
0619.19:20:10 <myrkraverk> and other sutff that I'd like to do with the same mechanism
0619.19:20:11 <hroptatyr> ah, yeah, that was my idea with these clustered emacsen
0619.19:20:17 <hroptatyr> yeah
0619.19:20:21 <myrkraverk> this would depend on a running emacs on the remote machine
0619.19:20:35 <hroptatyr> yepp
0619.19:20:51 <myrkraverk> I was thinking of moving compiled lisp back and forth
0619.19:21:05 <myrkraverk> perhaps I can do that with a module
0619.19:21:21 <myrkraverk> but I do no know enough about encryption to make it secure
0619.19:21:22 <hroptatyr> hehe, if we got that to work, we'd have an multi-threaded emacs ;P
0619.19:21:39 <myrkraverk> hmm, not multi threaded, a multi process
0619.19:21:44 <hroptatyr> actually all my approaches require re-evaluating the lisp at the client side
0619.19:22:58 <hroptatyr> hm, i'd know the encryption to make it secure ;)
0619.19:23:01 <myrkraverk> implementation wise, multi threaded and multi process are very different, though they would probable seem the same to the user (sort of)
0619.19:23:05 <myrkraverk> heh ;)
0619.19:23:08 <myrkraverk> nice ;)
0619.19:23:19 <myrkraverk> I was even thinking of the same ssh-ish mechanism cvs uses
0619.19:23:33 <myrkraverk> oh (re-evaluation)
0619.19:23:39 <hroptatyr> this pserver thingie? :o
0619.19:23:41 <myrkraverk> that would require yet another compilation
0619.19:23:52 <Procton> what benefits would a clustured emacsen give? I don't quite see a reason for such a feature.
0619.19:23:55 <hroptatyr> yepp
0619.19:24:02 <myrkraverk> (the CVS_RSH=ssh thingy, dunno about pserver, but perhaps)
0619.19:24:24 <myrkraverk> Procton: try using it on a 32 meg ram machine
0619.19:24:40 <Procton> ah.
0619.19:24:51 <hroptatyr> myrkraverk: ah :) ... hm, one could use ssl network streams
0619.19:24:51 <myrkraverk> Procton: also, when using many workstations around the house, this could be beneficial
0619.19:25:16 <myrkraverk> k, but I'll let you think about encription, once we have a protocol ;)
0619.19:25:38 <Procton> I'd see a reason for cpu intense applications, like compilers and such, but for emacs... the communication is probably heavier than actually the stuff that does things.
0619.19:25:39 <hroptatyr> hehe :)
0619.19:26:00 <myrkraverk> Procton: no neccesarily
0619.19:26:02 <myrkraverk> not
0619.19:26:15 <hroptatyr> Procton: yeah, but it could be really simple things, like watching the modeline of other emacs sessions
0619.19:26:57 <Procton> so... it is more like hooking up to already running emacsen, rather than actually a clustered solution?
0619.19:27:19 <hroptatyr> hm, yeah
0619.19:27:29 <Procton> k... I see a reason for that.
0619.19:27:49 <myrkraverk> Procton: remember that there exists code browsers, parsers and stuff in lisp
0619.19:28:19 * Procton pets vhdl mode.
0619.19:29:10 <myrkraverk> I see no reason to re-parse a complex and big file, by just opening it up on 2 or 3 workstations
0619.19:29:22 <hroptatyr> yeah
0619.19:29:30 <Procton> meaning?
0619.19:29:35 <myrkraverk> (though that could be handled by dbms caches for the parser)
0619.19:29:44 <Procton> you parse it on one workstation, and sends what to the others?
0619.19:30:04 <myrkraverk> Procton: waiting 20-30 sec to parse a file on a slow workstation, when it's already open in a running emacs on my fast pc?
0619.19:30:19 <myrkraverk> Procton: I just display the buffer on the other emacs
0619.19:30:21 <Procton> but... what would you send between the two?
0619.19:30:28 <myrkraverk> Procton: like another x frame
0619.19:30:40 <myrkraverk> Procton: that's what were thinking about
0619.19:30:51 <hroptatyr> yeah, maximally the amount of data of the X protocol ;)
0619.19:31:17 * Procton shrugs.
0619.19:31:31 <myrkraverk> I'd prefer to front-end the self-insert for one, and get compiled lisp for the display changes to the buffer back
0619.19:31:34 <Procton> X is nice and all, but is quite a heavy protocol.
0619.19:31:48 <myrkraverk> Procton: also, not all of use have (or use) X
0619.19:32:11 <hroptatyr> myrkraverk: yepp, nice approach
0619.19:32:26 <myrkraverk> hroptatyr: but does require more thought
0619.19:32:59 <myrkraverk> there would have to be a local buffer, to recieve the self-insert
0619.19:33:21 <myrkraverk> but perhaps we can do a back end with a module?
0619.19:33:33 <myrkraverk> (this requires some research)
0619.19:34:08 <hroptatyr> hm, yeah my original idea was also to envelope some local stuff ... send it to the other client and let that do the evaluation
0619.19:34:37 <hroptatyr> myrkraverk: d'you have a berlios account?
0619.19:34:55 <myrkraverk> hroptatyr: no
0619.19:35:39 <Procton> I'm just a bit sceptic regarding clustured solutions. It is not always more effective.
0619.19:35:45 <myrkraverk> for stuff like indent-line, I'm not sure what I'd like to do
0619.19:36:15 <myrkraverk> Procton: who needs effectiveness? this is xemacs!
0619.19:36:25 <Procton> hehe...
0619.19:36:46 <myrkraverk> perhaps send the compiled function call?
0619.19:36:57 <hroptatyr> Procton: hm, personally I do have effective machines everywhere here, but for example to change my current playing track from this emacs session here is kinda hard
0619.19:37:50 <hroptatyr> myrkraverk: why not making it multilayered ... this would be one possibility ... the other is to indent on the client machine and send some diff of the files to the remote
0619.19:37:52 <Procton> understood.
0619.19:38:25 <myrkraverk> hroptatyr: hmmmm, that would require the mode to be running on the local machine of course
0619.19:39:12 <hroptatyr> hm, yeah and no, that's why the protocol should offer the choice ;)
0619.19:39:32 <myrkraverk> hroptatyr: sending an (indent-line point mark) [or something] [nevermind]
0619.19:39:43 <myrkraverk> hmm, yeah
0619.19:39:53 <myrkraverk> or different methods of diplaying the file
0619.19:40:03 <hroptatyr> yepp :)
0619.19:40:08 <myrkraverk> find-file-remote or display-remote-buffer
0619.19:41:41 <hroptatyr> yeah, exactly
0619.19:42:23 <hroptatyr> the question is how far should that be implemented ... my idea was to provide rather just the abstract layer
0619.19:42:38 <myrkraverk> (for f-f-remote, the user could be offered [or not] the choice, if the file is already open)
0619.19:43:11 <myrkraverk> hroptatyr: hmmm, what abstract layer?
0619.19:43:36 <myrkraverk> hroptatyr: I was thinking, if I'm upto it, to write a module for passing compiled lisp back and forth for evaluation
0619.19:43:44 <hroptatyr> yeah, that could be done via sniffing what the remote emacs (message ...)'s or (error ...)'s and display that to the local machine
0619.19:44:08 <hroptatyr> myrkraverk: yeah, that's the abstract layer almost
0619.19:44:16 <myrkraverk> ah, k
0619.19:44:17 <hroptatyr> or at least an important part
0619.19:44:21 <myrkraverk> but I don't like the idea of sniffing
0619.19:44:49 <myrkraverk> this might require some sort of a feature of the lisp engine too
0619.19:44:54 <hroptatyr> the other is how to encapsulate local code for remote evaluation on the network level and how to make the connections secure
0619.19:45:31 <hroptatyr> hm, as far as my research has shown actually not more than implemented already in the edebug package
0619.19:46:01 <myrkraverk> yeah, that's something I'd need to do some research on, I've forgotten most of my network knowhow, and I've always worked with some abstraction of telnet
0619.19:46:16 <myrkraverk> hmmm
0619.19:46:28 <myrkraverk> I haven't looked at edebug, but yes, could be
0619.19:46:41 <hroptatyr> hehe, yeah ... mostly do i ... the first approach is a simple telnet -> netcat communication
0619.19:46:51 <hroptatyr> some other was running a local irc server
0619.19:46:55 <myrkraverk> netcat?
0619.19:46:56 <hroptatyr> ;)
0619.19:47:03 <myrkraverk> what's netcat?
0619.19:47:11 <myrkraverk> you have pets in your network?
0619.19:47:16 <hroptatyr> yepp, netcat can establish listening sockets ...
0619.19:47:17 <hroptatyr> lol
0619.19:47:26 <hroptatyr> it's like cat but for nettraffic
0619.19:47:27 <hroptatyr> ;)


---

Updated: 2005-12-30 15:40 (UTC)

Home / Index

Created by emacs-wiki.el