![]() Once the resources your job requires are free, the cluster will assign it to a worker node which will execute the job.Īnything written to the standard out and standard error will be written to the files you specified (if you did not specify these, the default is in your home directory).Įach job is allocated 1 core and about 8gb of RAM. ![]() The state column marks whether or not the job is running ( r), queued ( qw), or has failed with an error ( Eqw). The output from qstat shows information about all of the jobs currently queued or running on the grid. The scripts can read or write to files, run python or Java code, execute bash commands, etc.Īnything you can do in a normal bash script can also be done here.Īfter the script has been submitted, you should be able to see the job listed in the queue with the qstat command. The job bash scripts are identical to regular bash scripts except that it also includes extra arguments which are passed to the cluster.Īrguments about the name of the job, where the stdout should be written, how much memory it requires, etc., can all be included within the bash file on lines that start with #$.įor example, in the following hello.sh scriptĪny arguments you provide after the script name will be passed to the script as if you ran the script with sh. The bash script is submitted to the cluster, which puts it into the queue and runs the job when the required resources are free. Jobs on the NLP Grid take the form of a bash script. ![]() The NLP Grid uses the Sun Grid Engine (SGE) to schedule jobs and manage shared resources. If you have some data that you want to share with everyone on the NLP Grid, then it should be saved under /nlp/data/corpora.įor example, a lot of the LDC data is stored under /nlp/data/corpora/LDC. You can create your own directory and put your private code and data there. The recommended place to store your data is /nlp/data/. The NLP Grid has extra disks that are significantly larger than the SEAS home directories, but they are only accessible on the NLP Grid machines. Unfortunately, this means that the home directory is pretty restricted in terms of how much space you use. The home directory of the NLP Grid machines are the same ones that you access by logging into eniac, biglab, or a physical computer in a lab. Then, you can ssh into with your account information. To get access to the NLP Grid, email and cc the NLP faculty who is sponsoring your request. The purpose of this page is to (1) provide a new user with instructions to start using the NLP grid, (2) collect a cheatsheet of useful arguments to the cluster, and (3) provide some troubleshooting tips for common problems. In total, there are 11 machines, each with 64 cores and 500gb of RAM. It allows for large scale parallel processing that would be impossible to do on your own machine. The NLP Grid is a computing cluster shared among NLP researchers at Penn.
0 Comments
Leave a Reply. |
AuthorWrite something about yourself. No need to be fancy, just an overview. ArchivesCategories |