| Store | Cart

Re: [Tutor] When to use multiprocessing Managers?

From: James Chapman <jam...@uplinkzero.com>
Mon, 3 Mar 2014 11:45:40 +0000
Thanks for the explanation.

Is there any reason or case you can think of where on a single system
you would use the manager (proxied) queue over the multiprocessing
(piped) queue?

Actually I can probably answer this myself...

The manager could potentially be extended to do some kind of data
validation / error checking before submitting to the Queue. This could
be important if the data going into the Queue was for example, user
generated.

Hmm, yeah I'd say question answered. Thanks eryksun.


--
James


On 1 March 2014 16:48, eryksun <eryk...@gmail.com> wrote:
> On Fri, Feb 28, 2014 at 6:31 AM, James Chapman <jam...@uplinkzero.com> wrote:>>>> log_Q = multiprocessing.Queue()>> This is a Queue from multiprocessing.queues. It uses system resources> (e.g. a semaphore for the queue capacity) that can be shared with> processes on the same machine.>> A value `put` in a queue.Queue is available immediately:>>     >>> import queue>     >>> q1 = queue.Queue()>     >>> try: q1.put('value'); q1.get_nowait()>     ... except queue.Empty: 'empty'>     ...>     'value'>> On the other hand, a Queue from multiprocessing.queues writes to a> pipe using a background thread, so there can be a small delay:>>     >>> import multiprocessing as mp>     >>> q2 = mp.Queue()>     >>> try: q2.put('value'); q2.get_nowait()>     ... except queue.Empty: 'empty'>     ...>     'empty'>     >>> q2.get_nowait()>     'value'>>> or whether I create it like this:>>>> multimanager = multiprocessing.Manager()>> log_Q = multimanager.Queue()>> This is a queue.Queue wrapped by an AutoProxy. For example, its `get`> method calls _callmethod('get', *args, **kwds), which connects to the> manager, sends the request, and receives the result.>> The docs demonstrate using a manager with remote processes:>> http://docs.python.org/3/library/multiprocessing#using-a-remote-manager>> You can also proxy the Queue type from multiprocessing.queues. In that> case, remote processes use a proxy, but local processes can use the> queue directly.>>> Perhaps the manager would be important if I was writing to a Queue and>> expecting all threads to see that message?>> Only 1 thread will `get` the message.
_______________________________________________
Tutor maillist  -  Tut...@python.org
To unsubscribe or change subscription options:
https://mail.python.org/mailman/listinfo/tutor

Recent Messages in this Thread
James Chapman Feb 25, 2014 10:52 am
Steven DAprano Feb 25, 2014 10:45 pm
Danny Yoo Feb 25, 2014 10:55 pm
Steven DAprano Feb 25, 2014 11:05 pm
David Palao Feb 26, 2014 02:19 pm
James Chapman Feb 28, 2014 11:31 am
eryksun Mar 01, 2014 04:48 pm
James Chapman Mar 03, 2014 11:45 am
eryksun Mar 03, 2014 03:40 pm
Messages in this thread