Softpanorama

May the source be with you, but remember the KISS principle ;-)
Contents Bulletin Scripting in shell and Perl Network troubleshooting History Humor

MPI startup(): RLIMIT_MEMLOCK too small problem with SGE and Infiniband

The problem does not depend on which version of MPI you are using: it is observable both with Intel MPI 4.1, Melllanoix OpenMPI 1.6.4 and several others.

One sthing you can do is to  increase the RLIMIT_MEMLOCK limit to 32MB instead of just 32 or 64KB by adding the  command

ulimit -l 32000

to the sgeexecd init script.

But this does not work.

Please have a look into the forum thread: https://software.intel.com/en-us/forums/topic/382682

Putting

execd_params H_MEMORYLOCKED=UNLIMITED

does not work either in Univa SGE. But it works in Oracle SGE. 

 


Top updates

Bulletin Latest Past week Past month
Google Search


NEWS CONTENTS

Old News ;-)

Re [GE users] OpenMPI RLIMIT_MEMLOCK problem - Prentice Bisbal - net.sunsource.gridengine.users - MarkMail

atom feed5 messages in net.sunsource.gridengine.usersRe: [GE users] OpenMPI RLIMIT_MEMLOCK...
From Sent On Attachments
Prentice Bisbal Dec 3, 2008 12:54 pm
Prentice Bisbal Dec 3, 2008 2:06 pm
Prentice Bisbal Dec 3, 2008 2:49 pm
pollinger Dec 4, 2008 1:57 am
Prentice Bisbal Dec 4, 2008 6:34 am
Subject: Re: [GE users] OpenMPI RLIMIT_MEMLOCK problem
From: Prentice Bisbal (pren@ias.edu)
Date: Dec 3, 2008 2:49:30 pm
List: net.sunsource.gridengine.users
I think I found a bug in 6.2. When I set

execd_params H_MEMORYLOCKED=unlimited

and restart sge_execd on all my nodes, I get this:

$ qrsh ulimit -l 4

When I set

execd_params H_MEMORYLOCKED=UNLIMITED,

I get this

$ qrsh ulimit -l 5

If set H_MEMORYLOCKED to a numerical value, like this

execd_params H_MEMORYLOCKED=32M

I get the correct result:

qrsh ulimit -l 32768

Looks like the unlimited keyword is not defined properly in the code somewhere. Also, I have to restart sge_execd to get these changes to take effect. Is that the proper behavior? The documentation I linked to in my original post didn't explicitly say anything about restarting sge_execd after tinkering with these settings.

-- Prentice

Prentice Bisbal wrote:

I posted this earlier, but it never showed up on the list. Probably b/c I unsubscribed before SC08 and forget to resubscribe. If if the first attempt shows up, I apologize for posting twice.

I'm using SGE 6.2 with OpenMPI 1.2.8.

I just setup OpenMPI tight integration following the instructions here:

http://www.open-mpi.​org/faq/?category=ru​nning#run-n1ge-or-sg​e

I then defined my exec_params to set H_MEMLOCK=unlimited as described here:

http://gridengine.su​nsource.net/ds/viewM​essage.do?dsForumId=​38​ &dsMessageId=72405

Unfortunately, when I submit an MPI job, I still get MEMLOCK errors:

libibverbs: Warning: RLIMIT_MEMLOCK is 11162 bytes. This will severely limit memory registrations. libibverbs: Warning: RLIMIT_MEMLOCK is 11199 bytes. This will severely limit memory registrations. libibverbs: Warning: RLIMIT_MEMLOCK is 11181 bytes. This will severely limit memory registrations.

To make sure my parameters were took effect, I even stopped and restarted sge_execd on every compute node with no luck.

Any ideas? Relevant configuration information is below. Please note that in qconf I've tried

1. execd_params H_MEMORYLOCKED=ulimited 2. execd_params S_MEMORYLOCKED=ulimited 3. execd_params S_MEMORYLOCKED=ulimited H_MEMORYLOCKED=ulimited

All 3 had the same result.

Let me know if additional/complete config information would be helpful. I don't want to flood the list with unnecessary config information.

# qconf -sp orte pe_name orte slots 512 user_lists NONE xuser_lists NONE start_proc_args /bin/true stop_proc_args /bin/true allocation_rule $fill_up control_slaves TRUE job_is_first_task FALSE urgency_slots min accounting_summary FALSE

# qconf -sq all.q | grep pe_list pe_list make orte

# qconf -sconf | grep execd_params execd_params H_MEMORYLOCKED=unlimited S_MEMORYLOCKED=unlimited

$ more xhpl.sh #$ -S /bin/bash #$ -N xhpl #$ -pe orte 512 #$ -cwd #$ -V

MPI=/usr/local/openm​pi/gcc-4.1.2/x86_64/​ PATH=${MPI}/bin:$i{PATH} LD_LIBRARY_PATH=${MPI}/lib

mpirun ./xhpl

-- Prentice

------------------------------------------------------ http://gridengine.sunsource.net/ds/viewMessage.do?dsForumId=38&dsMessageId=90969

To unsubscribe from this discussion, e-mail:
[user@gridengine.sunsource.net].

MPI startup() ofa fabric is not available

James Tullos (Intel)

Fri, 03/29/2013 - 06:52

Hi John,

Let me check with the developers for details about that. What do you get from "env | grep I_MPI"?

Sincerely,
James Tullos
Technical Consulting Engineer
Intel® Cluster Tools

Top

James Tullos (Intel)

Fri, 03/29/2013 - 06:58

Hi John,

Also, can you please send the output from a run with I_MPI_DEBUG=2? This will show more details about the fabric selection process.

Sincerely,
James Tullos
Technical Consulting Engineer
Intel® Cluster Tools

Top

James Tullos (Intel)

Fri, 03/29/2013 - 07:53

Hi John,

The developers will also need to know your OS/distribution, the OFED version you are using, and how you are launching the application.

Sincerely,
James Tullos
Technical Consulting Engineer
Intel® Cluster Tools

Top

John Cavanaugh

Mon, 04/01/2013 - 08:36

Thanks for your responses. My I_MPI environment variables are:

> I_MPI_INC=/cray/css/iaa/mpi_images/impi/4.1.0.024/intel64/intel64/include
> I_MPI_F77=/opt/intel/composerxe-2011.5.220/bin/intel64/ifort
> I_MPI_FABRICS=shm:ofa
> I_MPI_LIB=/cray/css/iaa/mpi_images/impi/4.1.0.024/intel64/intel64/lib
> I_MPI_CC=/opt/intel/composerxe-2011.5.220/bin/intel64/icc
> I_MPI_DEBUG=2

Output from the job is:

> [1] MPI startup(): RLIMIT_MEMLOCK too small
> [1] MPI startup(): ofa fabric is not available and fallback fabric is not enabled
> [0] MPI startup(): ofa fabric is not available and fallback fabric is not enabled
> [0] MPI startup(): RLIMIT_MEMLOCK too small

I'm running this on a Cray XC30 running Linux SLESS 11 SP2. The release is 3.0.42. I'm launching the job with mpirun. The runtime environment is provided by CCM, a Cray product that provides a cluster-like execution environment on Cray compute nodes. The OFA layer is provided by IAA, another Cray product that is a library that provides an IB verbs interface but does data transfer directly across the Cray high-speed network.

As I said in my original post, this works with 4.0.2, but fails with 4.1.0. I suspect there's some envirnment variable I don't know about, or perhaps I'm setting something incorrectly.

Thanks.

Top

James Tullos (Intel)

Mon, 04/01/2013 - 08:41

Hi John,

What is the output from ulimit -a?

Sincerely,
James Tullos
Technical Consulting Engineer
Intel® Cluster Tools

Top

John Cavanaugh

Mon, 04/01/2013 - 08:55

Ulimit -a gives:

> core file size (blocks, -c) unlimited
> data seg size (kbytes, -d) unlimited
> scheduling priority (-e) 0
> file size (blocks, -f) unlimited
> pending signals (-i) 257676
> max locked memory (kbytes, -l) 64
> max memory size (kbytes, -m) unlimited
> open files (-n) 32768
> pipe size (512 bytes, -p) 8
> POSIX message queues (bytes, -q) 819200
> real-time priority (-r) 0
> stack size (kbytes, -s) unlimited
> cpu time (seconds, -t) unlimited
> max user processes (-u) 257676
> virtual memory (kbytes, -v) 26395360
> file locks (-x) unlimited

I saw the "RLIMIT_MEMLOCK too small" and tried 'ulimit -l unlimited', but it didn't make a difference.

Top

John Cavanaugh

Mon, 04/01/2013 - 09:15

Output from 'ulmit -a' is:

> core file size (blocks, -c) unlimited
> data seg size (kbytes, -d) unlimited
> scheduling priority (-e) 0
> file size (blocks, -f) unlimited
> pending signals (-i) 257676
> max locked memory (kbytes, -l) 64
> max memory size (kbytes, -m) unlimited
> open files (-n) 32768
> pipe size (512 bytes, -p) 8
> POSIX message queues (bytes, -q) 819200
> real-time priority (-r) 0
> stack size (kbytes, -s) unlimited
> cpu time (seconds, -t) unlimited
> max user processes (-u) 257676
> virtual memory (kbytes, -v) 26395360
> file locks (-x) unlimited

I noticed the "RLIMIT_MEMLOCK too small" messge and tried 'ulimit -c unlimited -l unlimited' but it didn't make any difference (the above output is from after the ulmit command).

Top

James Tullos (Intel)

Mon, 04/01/2013 - 11:40

Hi John,

Using

view sourceprint?
1 ulimit -l unlimited
should have set the locked memory limit to unlimited (which is the one I'm concerned about). Is there anything in your /etc/security/limits.conf file that is putting a hard limit on locked memory?

Sincerely,
James Tullos
Technical Consulting Engineer
Intel® Cluster Tools

Top

John Cavanaugh

Mon, 04/01/2013 - 11:53

It doesn't look like there's anything in /etc/security/limits.conf. The whole file is comments and empty lines.

Top

John Cavanaugh

Mon, 04/01/2013 - 11:56

I get an error when I try to use 'ulimit -l':

> ulimit: max locked memory: cannot modify limit: Operation not permitted

Top

James Tullos (Intel)

Mon, 04/01/2013 - 12:08

Hi John,

Check in /etc/security/limit.d/ for any additional files. You should be able to modify the locked memory limit. Are you trying this as root or as a standard user?

Sincerely,
James Tullos
Technical Consulting Engineer
Intel® Cluster Tools

Top

John Cavanaugh

Mon, 04/01/2013 - 12:18

/etc/security/limit.d/ doesn't exist in my environment.

I'm trying to run as a standard user. It appears that I can reduce the max locked memory, but not increase it:

> jdc@nid00012 ~ $ ulimit -l 16
> jdc@nid00012 ~ $ ulimit -l 32
> -bash: ulimit: max locked memory: cannot modify limit: Operation not permittedjdc@nid00012 ~ $ ulimit -a
> jdc@nid00012 ~ $ ulimit -a
> ...
> max locked memory (kbytes, -l) 16

How big do I need max locked memory to be?

Top

James Tullos (Intel)

Mon, 04/01/2013 - 12:24

Hi John,

We recommend setting it to unlocked, but it is dependent on your application. You very likely have a hard limit set somewhere on your system, a standard user cannot go higher than the hard limit. If you can't get the memory limit higher, let's try a different approach. There is a basic test program included with the Intel® MPI Library. It is in the test/ subfolder of the installation. Compile any one of the source files present, and try running that program. The memory usage is extremely low (simply a Hello World program), and it should run with a lower memory limit.

Sincerely,
James Tullos
Technical Consulting Engineer
Intel® Cluster Tools

Top

John Cavanaugh

Tue, 04/02/2013 - 09:42

The "Hello, world" test program runs over TCP, but fails over DAPL and OFA, just as IMB did.

I have a query out about changing the max locked memory limit. Forgive me for asking this, but do we know that the memory limit is the reason that fabric detection is failing?

Thanks.

Top

James Tullos (Intel)

Tue, 04/02/2013 - 11:11

Hi John,

I'm not certain that the memory limit is the cause of the problem. But the memory limit is definitely a problem, otherwise the RLIMIT_MEMLOCK too small messages would not be shown. I'm checking with the developers for more information.

Sincerely,
James Tullos
Technical Consulting Engineer
Intel® Cluster Tools

Top

James Tullos (Intel)

Tue, 04/02/2013 - 11:38

Hi John,

I've checked with the developers. The locked memory limit will need to be at least 32 MB in order to run. This is a requirement set in our code. Please set it to at least 32 MB and try again.

Sincerely,
James Tullos
Technical Consulting Engineer
Intel® Cluster Tools

Top

John Cavanaugh

Tue, 04/02/2013 - 12:52

I found a (Cray-specific) way to raise the locked memory limit and both the "Hello, world" test and IMB run under 4.1.0 now.

Thanks for your help.

Top

James Tullos (Intel)

Tue, 04/02/2013 - 12:54

Hi John,

Great! I'm glad everything is working now.

Sincerely,
James Tullos
Technical Consulting Engineer
Intel® Cluster Tools



Etc

FAIR USE NOTICE This site contains copyrighted material the use of which has not always been specifically authorized by the copyright owner. We are making such material available in our efforts to advance understanding of environmental, political, human rights, economic, democracy, scientific, and social justice issues, etc. We believe this constitutes a 'fair use' of any such copyrighted material as provided for in section 107 of the US Copyright Law. In accordance with Title 17 U.S.C. Section 107, the material on this site is distributed without profit exclusivly for research and educational purposes.   If you wish to use copyrighted material from this site for purposes of your own that go beyond 'fair use', you must obtain permission from the copyright owner. 

ABUSE: IPs or network segments from which we detect a stream of probes might be blocked for no less then 90 days. Multiple types of probes increase this period.  

Society

Groupthink : Two Party System as Polyarchy : Corruption of Regulators : Bureaucracies : Understanding Micromanagers and Control Freaks : Toxic Managers :   Harvard Mafia : Diplomatic Communication : Surviving a Bad Performance Review : Insufficient Retirement Funds as Immanent Problem of Neoliberal Regime : PseudoScience : Who Rules America : Neoliberalism  : The Iron Law of Oligarchy : Libertarian Philosophy

Quotes

War and Peace : Skeptical Finance : John Kenneth Galbraith :Talleyrand : Oscar Wilde : Otto Von Bismarck : Keynes : George Carlin : Skeptics : Propaganda  : SE quotes : Language Design and Programming Quotes : Random IT-related quotesSomerset Maugham : Marcus Aurelius : Kurt Vonnegut : Eric Hoffer : Winston Churchill : Napoleon Bonaparte : Ambrose BierceBernard Shaw : Mark Twain Quotes

Bulletin:

Vol 25, No.12 (December, 2013) Rational Fools vs. Efficient Crooks The efficient markets hypothesis : Political Skeptic Bulletin, 2013 : Unemployment Bulletin, 2010 :  Vol 23, No.10 (October, 2011) An observation about corporate security departments : Slightly Skeptical Euromaydan Chronicles, June 2014 : Greenspan legacy bulletin, 2008 : Vol 25, No.10 (October, 2013) Cryptolocker Trojan (Win32/Crilock.A) : Vol 25, No.08 (August, 2013) Cloud providers as intelligence collection hubs : Financial Humor Bulletin, 2010 : Inequality Bulletin, 2009 : Financial Humor Bulletin, 2008 : Copyleft Problems Bulletin, 2004 : Financial Humor Bulletin, 2011 : Energy Bulletin, 2010 : Malware Protection Bulletin, 2010 : Vol 26, No.1 (January, 2013) Object-Oriented Cult : Political Skeptic Bulletin, 2011 : Vol 23, No.11 (November, 2011) Softpanorama classification of sysadmin horror stories : Vol 25, No.05 (May, 2013) Corporate bullshit as a communication method  : Vol 25, No.06 (June, 2013) A Note on the Relationship of Brooks Law and Conway Law

History:

Fifty glorious years (1950-2000): the triumph of the US computer engineering : Donald Knuth : TAoCP and its Influence of Computer Science : Richard Stallman : Linus Torvalds  : Larry Wall  : John K. Ousterhout : CTSS : Multix OS Unix History : Unix shell history : VI editor : History of pipes concept : Solaris : MS DOSProgramming Languages History : PL/1 : Simula 67 : C : History of GCC developmentScripting Languages : Perl history   : OS History : Mail : DNS : SSH : CPU Instruction Sets : SPARC systems 1987-2006 : Norton Commander : Norton Utilities : Norton Ghost : Frontpage history : Malware Defense History : GNU Screen : OSS early history

Classic books:

The Peter Principle : Parkinson Law : 1984 : The Mythical Man-MonthHow to Solve It by George Polya : The Art of Computer Programming : The Elements of Programming Style : The Unix Hater’s Handbook : The Jargon file : The True Believer : Programming Pearls : The Good Soldier Svejk : The Power Elite

Most popular humor pages:

Manifest of the Softpanorama IT Slacker Society : Ten Commandments of the IT Slackers Society : Computer Humor Collection : BSD Logo Story : The Cuckoo's Egg : IT Slang : C++ Humor : ARE YOU A BBS ADDICT? : The Perl Purity Test : Object oriented programmers of all nations : Financial Humor : Financial Humor Bulletin, 2008 : Financial Humor Bulletin, 2010 : The Most Comprehensive Collection of Editor-related Humor : Programming Language Humor : Goldman Sachs related humor : Greenspan humor : C Humor : Scripting Humor : Real Programmers Humor : Web Humor : GPL-related Humor : OFM Humor : Politically Incorrect Humor : IDS Humor : "Linux Sucks" Humor : Russian Musical Humor : Best Russian Programmer Humor : Microsoft plans to buy Catholic Church : Richard Stallman Related Humor : Admin Humor : Perl-related Humor : Linus Torvalds Related humor : PseudoScience Related Humor : Networking Humor : Shell Humor : Financial Humor Bulletin, 2011 : Financial Humor Bulletin, 2012 : Financial Humor Bulletin, 2013 : Java Humor : Software Engineering Humor : Sun Solaris Related Humor : Education Humor : IBM Humor : Assembler-related Humor : VIM Humor : Computer Viruses Humor : Bright tomorrow is rescheduled to a day after tomorrow : Classic Computer Humor

The Last but not Least


Copyright © 1996-2016 by Dr. Nikolai Bezroukov. www.softpanorama.org was created as a service to the UN Sustainable Development Networking Programme (SDNP) in the author free time. This document is an industrial compilation designed and created exclusively for educational use and is distributed under the Softpanorama Content License.

The site uses AdSense so you need to be aware of Google privacy policy. You you do not want to be tracked by Google please disable Javascript for this site. This site is perfectly usable without Javascript.

Original materials copyright belong to respective owners. Quotes are made for educational purposes only in compliance with the fair use doctrine.

FAIR USE NOTICE This site contains copyrighted material the use of which has not always been specifically authorized by the copyright owner. We are making such material available to advance understanding of computer science, IT technology, economic, scientific, and social issues. We believe this constitutes a 'fair use' of any such copyrighted material as provided by section 107 of the US Copyright Law according to which such material can be distributed without profit exclusively for research and educational purposes.

This is a Spartan WHYFF (We Help You For Free) site written by people for whom English is not a native language. Grammar and spelling errors should be expected. The site contain some broken links as it develops like a living tree...

You can use PayPal to make a contribution, supporting development of this site and speed up access. In case softpanorama.org is down you can use the at softpanorama.info

Disclaimer:

The statements, views and opinions presented on this web page are those of the author (or referenced source) and are not endorsed by, nor do they necessarily reflect, the opinions of the author present and former employers, SDNP or any other organization the author may be associated with. We do not warrant the correctness of the information provided or its fitness for any purpose.

Last modified: October 27, 2016