Seqanswers Leaderboard Ad

Collapse

Announcement

Collapse
No announcement yet.
X
 
  • Filter
  • Time
  • Show
Clear All
new posts

  • #61
    I made a lot of people angry with Ray today....

    Hi seb567,

    So, I had Ray 1.2.0 installed on our cluster, complied with the same with Intel Compiler. The job starts to run, then is crashing.... Apparently today I took down 32 compute nodes according to the angry email I received from IT...

    They sent me the following information about the job, I am told the problem is the huge amount of swap space I was using -- nearly 1TB!

    Code:
    Req[0]  TaskCount: 256  Partition: anon
    Utilized Resources Per Task:  PROCS: 120.18  MEM: 2596M  SWAP: 881G
    Avg Util Resources Per Task:  PROCS: 120.18
    Max Util Resources Per Task:  PROCS: 237.11  MEM: 2596M  SWAP: 881G
    Average Utilized Memory: 1641.30 MB
    Average Utilized Procs: 48473.59
    NodeSet=ONEOF:FEATURE:awesometown
    NodeAccess: SINGLEJOB
    NodeCount:  32
    Here is the tail of my outfile:

    Code:
    Rank 230 is adding ingoing edges (reverse complement) 300001/1095111
    [[18226,1],230][btl_openib_component.c:3224:handle_wc] from s54-5.local to: s54-15 error polling LP CQ with status RETRY EXCEEDED ERROR status number 12 for wr_id 246144128 opcode 0  vendor error 129 qp_idx 2
    [s56-6.local:00534] 33 more processes have sent help message help-mpi-btl-openib.txt / pp retry exceeded
    [s56-6.local:00534] Set MCA parameter "orte_base_help_aggregate" to 0 to see all help / error messages
    [[18226,1],232][btl_openib_component.c:3224:handle_wc] from s54-4.local to: s55-8 error polling HP CQ with status WORK REQUEST FLUSHED ERROR status number 5 for wr_id 13189504 opcode 32767  vendor error 244 qp_idx 0
    [[18226,1],173][btl_openib_component.c:3224:handle_wc] from s54-12.local to: s55-6 error polling HP CQ with status WORK REQUEST FLUSHED ERROR status number 5 for wr_id 16690688 opcode 32767  vendor error 244 qp_idx 0
    [[18226,1],237][btl_openib_component.c:3224:handle_wc] from s54-4.local to: s54-12 error polling LP CQ with status RETRY EXCEEDED ERROR status number 12 for wr_id 121666816 opcode 32767  vendor error 129 qp_idx 2
    [s56-6.local:00534] 1 more process has sent help message help-mpi-btl-openib.txt / pp retry exceeded
    [[18226,1],234][btl_openib_component.c:3224:handle_wc] from s54-4.local to: s56-3 error polling HP CQ with status WORK REQUEST FLUSHED ERROR status number 5 for wr_id 13992832 opcode 32767  vendor error 244 qp_idx 0
    [[18226,1],175][btl_openib_component.c:3224:handle_wc] from s54-12.local to: s55-13 error polling LP CQ with status RETRY EXCEEDED ERROR status number 12 for wr_id 113500032 opcode 32767  vendor error 129 qp_idx 2
    [s56-6.local:00534] 1 more process has sent help message help-mpi-btl-openib.txt / pp retry exceeded
    [[18226,1],172][btl_openib_component.c:3224:handle_wc] from s54-12.local to: s55-13 error polling LP CQ with status RETRY EXCEEDED ERROR status number 12 for wr_id 117999232 opcode 32767  vendor error 129 qp_idx 2
    [[18226,1],170][btl_openib_component.c:3224:handle_wc] from s54-12.local to: s55-13 error polling LP CQ with status RETRY EXCEEDED ERROR status number 12 for wr_id 59806336 opcode 1  vendor error 129 qp_idx 2
    [s56-6.local:00534] 2 more processes have sent help message help-mpi-btl-openib.txt / pp retry exceeded
    [[18226,1],238][btl_openib_component.c:3224:handle_wc] from s54-4.local to: s55-16 error polling HP CQ with status WORK REQUEST FLUSHED ERROR status number 5 for wr_id 17854848 opcode 128  vendor error 244 qp_idx 0
    [[18226,1],215][btl_openib_component.c:3224:handle_wc] from s54-7.local to: s54-12 error polling LP CQ with status RETRY EXCEEDED ERROR status number 12 for wr_id 115086208 opcode 0  vendor error 129 qp_idx 2
    [s56-6.local:00534] 1 more process has sent help message help-mpi-btl-openib.txt / pp retry exceeded
    [[18226,1],197][btl_openib_component.c:3224:handle_wc] from s54-9.local to: s54-12 error polling LP CQ with status RETRY EXCEEDED ERROR status number 12 for wr_id 102530048 opcode 32767  vendor error 129 qp_idx 2
    [s56-6.local:00534] 1 more process has sent help message help-mpi-btl-openib.txt / pp retry exceeded
    [[18226,1],199][btl_openib_component.c:3224:handle_wc] from s54-9.local to: s54-12 error polling LP CQ with status RETRY EXCEEDED ERROR status number 12 for wr_id 61565696 opcode 32767  vendor error 129 qp_idx 2
    [s56-6.local:00534] 1 more process has sent help message help-mpi-btl-openib.txt / pp retry exceeded
    [[18226,1],171][btl_openib_component.c:3224:handle_wc] from s54-12.local to: s54-15 error polling HP CQ with status WORK REQUEST FLUSHED ERROR status number 5 for wr_id 9502080 opcode 128  vendor error 244 qp_idx 0
    [[18226,1],196][btl_openib_component.c:3224:handle_wc] from s54-9.local to: s54-2 error polling HP CQ with status WORK REQUEST FLUSHED ERROR status number 5 for wr_id 13076352 opcode 128  vendor error 244 qp_idx 0
    [[18226,1],169][btl_openib_component.c:3224:handle_wc] from s54-12.local to: s55-8 error polling HP CQ with status WORK REQUEST FLUSHED ERROR status number 5 for wr_id 21503744 opcode 128  vendor error 244 qp_idx 0
    [[18226,1],174][btl_openib_component.c:3224:handle_wc] from s54-12.local to: s56-6 error polling HP CQ with status WORK REQUEST FLUSHED ERROR status number 5 for wr_id 11039872 opcode 128  vendor error 244 qp_idx 0
    [[18226,1],194][btl_openib_component.c:3224:handle_wc] from s54-9.local to: s55-8 error polling HP CQ with status WORK REQUEST FLUSHED ERROR status number 5 for wr_id 12326400 opcode 32767  vendor error 244 qp_idx 0
    =>> PBS: job killed: node 26 (s54-7) requested job terminate, 'EOF' (code 1099) - internal or network failure attempting to communicate with sister MOM's
    mpirun: abort is already in progress...hit ctrl-c again to forcibly terminate
    
    15 total processes killed (some possibly by mpirun during cleanup)
    So, now IT is really angry with me, I'm kinda proud of myself, but I still have a genome to try to assemble. What do you suggest?

    Comment


    • #62
      Hi !



      error polling HP CQ with status WORK REQUEST FLUSHED ERROR

      Seems to be an Open-MPI issue.



      SWAP: 881G


      Also, it is a bad idea to have 1 terabyte of swap memory, really.





      How many MPI ranks do you have ?




      How much physical memory is available on your cluster for your job ?




      How many files / reads do you have ?



      What is the Ray command you used ?



      I wrote a post on my blog about Ray and large datasets:

      I scale Ray on 512 cores and 1536 gigabytes of distributed memory on a human genome. Still doing some tests though before Christmas.

      Ray is a de novo de Bruijn genome assembler that uses message passing interface. Sébastien Boisvert reports the memory utilization for Illu...

      Comment


      • #63
        Hi,

        Ray 1.2.0 crashes here, and I mean segfaults:

        ray.err:

        Loaded openMPI 1.4.2, compiled with intel11.1 (found in /opt/openmpi/1.4.2intel11.1/)
        [q241:26410] *** Process received signal ***
        [q241:26410] Signal: Segmentation fault (11)
        [q241:26410] Signal code: Address not mapped (1)
        [q241:26410] Failing at address: 0x9
        [q241:26410] [ 0] /lib64/libpthread.so.0 [0x2b51e6679b10]
        [q241:26410] [ 1] /bubo/home/h12/pallol/glob/sandbox/Ray-1.2.0/code/Ray(_ZN6Vertex15addOutgoingEdgeEmiP1\
        1MyAllocator+
        [q241:26410] [ 2] /bubo/home/h12/pallol/glob/sandbox/Ray-1.2.0/code/Ray(_ZN16MessageProcessor23call_TAG_\
        OUT_EDGES_DAT
        [q241:26410] [ 3] /bubo/home/h12/pallol/glob/sandbox/Ray-1.2.0/code/Ray(_ZN16MessageProcessor14processMe\
        ssageEP7Messa
        [q241:26410] [ 4] /bubo/home/h12/pallol/glob/sandbox/Ray-1.2.0/code/Ray(_ZN7Machine5startEv+0x1464) [0x4\
        31c04]
        [q241:26410] [ 5] /bubo/home/h12/pallol/glob/sandbox/Ray-1.2.0/code/Ray(main+0x89) [0x45a739]
        [q241:26410] [ 6] /lib64/libc.so.6(__libc_start_main+0xf4) [0x2b51e68a3994]
        [q241:26410] [ 7] /bubo/home/h12/pallol/glob/sandbox/Ray-1.2.0/code/Ray(__gxx_personality_v0+0xe9) [0x41\
        7289]
        [q241:26410] *** End of error message ***
        --------------------------------------------------------------------------
        mpirun noticed that process rank 144 with PID 26410 on node q241 exited on signal 11 (Segmentation fault\
        ).
        --------------------------------------------------------------------------
        slurmd[q75]: *** JOB 220623 CANCELLED AT 2010-12-16T01:07:13 DUE TO NODE FAILURE ***
        mpirun: abort is already in progress...hit ctrl-c again to forcibly terminate
        I have 16 lanes of illumina data (about 500M 90bp PE reads).
        I call Ray simply with:

        mpirun -np 160 ~/sandbox/Ray-1.2.0/code/Ray -p l_1_1.fastq l_1_2.fastq -p l_2_1.fastq...


        cheers
        pallo

        Comment


        • #64
          Hi pallo,

          Can you provide additional information (see my questions above) ?

          (You can send me an email too.)


          Otherwise it is difficult for me to rule out the problem.


          Thank you !

          Comment


          • #65
            Hi seb567

            Good news, looks like it works with gcc 4.4 + openmpi 1.4.2 (still ran out of memory though, but I will try a bigger set of nodes next time).

            I dont think we need to debug the Intel compiled version, but if you want debug data, Ill provide as much as I can.

            Which questions are you referring to? The ones you posted to caddymob?

            Most of this is in my post: the data is ca 500M reads of illumina PE data on 16 lanes plus 3 runs of 454 (total 16x2 + 3 = 35 files). The nodes are 20x8core so Im running mpirun 160. Each node has 24G mem + 24G swap.

            cheers

            Comment


            • #66
              @pallo: sorry, I mixed caddymob with you.

              I will do tests with the Intel compiler myself.

              Comment


              • #67
                Originally posted by pallo View Post
                I dont think we need to debug the Intel compiled version, but if you want debug data, Ill provide as much as I can.
                Have you compiled other biotools with the intel compilar and compared performance against gcc?
                -drd

                Comment


                • #68
                  Originally posted by drio View Post
                  Have you compiled other biotools with the intel compilar and compared performance against gcc?
                  hi drio,

                  Nope, I dont have any benchmarks...
                  Its just SOP here to try to compile with the "native" intel and then gcc if intel binaries fail.

                  Comment


                  • #69
                    next steps...

                    Originally posted by seb567 View Post

                    error polling HP CQ with status WORK REQUEST FLUSHED ERROR

                    Seems to be an Open-MPI issue.
                    Maybe .. our system was under high load and the time and the infinband links on the LUSTRE file system may have been saturated, compounding the problem. Nevertheless I have been warned not to try again until I fix the swap issue or risk losing my cluster account. IT is really mad at me for this


                    Originally posted by seb567 View Post
                    SWAP: 881G Also, it is a bad idea to have 1 terabyte of swap memory, really.
                    Agreed, but this is something Ray did, I do not, to my knowledge have a way to specify how much swap to use. What can I do to prevent this?

                    Originally posted by seb567 View Post
                    How many MPI ranks do you have ?
                    What is the Ray command you used ?
                    How much physical memory is available on your cluster for your job ?
                    I was using 256 cores -- our system has 8-core nodes, each with 24GB of RAM.

                    My command:
                    Code:
                    ###Parameterized PBS Script ####
                    #PBS -S /bin/bash
                    #PBS -N LUNDE.Ray4
                    #PBS -l nodes=256
                    #PBS -l walltime=25:00:00
                    #PBS -q normal
                    #PBS -j oe
                    #PBS -o LUNDE.Ray4.o
                    #PBS -M ---redacted---
                    #PBS -m abe
                    
                    wd=/scratch/myfiles/LUNDE_ASSEMBLE/
                    cd $wd
                    
                    use intel-openmpi-1.4.2
                    use Ray-1.2.0
                    
                    mpirun -np 256 Ray -p $wd\Lunde_1.fastq $wd\Lunde_2.fastq -o Lunde-contigs
                    Originally posted by seb567 View Post
                    How many files / reads do you have ?
                    I have 2 FASTQ files, one for the forward read, one for the reverse. I have 140,174,250 paired reads, 105mers. Would it be better to split these up into smaller chunks or does that matter?

                    Thanks for your help!

                    Comment


                    • #70
                      Dear caddymob,

                      I did some investigative work for you this morning.



                      You have 140 174 250 paired reads, each having a length of 105.

                      Would it be better to split these up into smaller chunks or does that matter?
                      Splitting the files would change nothing since Ray loads sequences in a lazy manner.



                      You use 256 MPI ranks, which are mapped on 8-core nodes, each having 16 GiB of memory.
                      You use 32 such nodes.

                      You have 512 GiB of distributed physical memory.



                      In modern operating systems, a program's addresses are virtual. If the computer has 2 GiB of random-access memory (RAM) and 100 GiB of swap memory,
                      then a program can allocate more than 2 GiB for its use. However, if the heap size (utilized memory) exceeds the random-access memory size, page faults occur.

                      "A page fault is a trap to the software raised by the hardware when a program accesses a page that is mapped in the virtual address space, but not loaded in physical memory."


                      It is followed by paging, 'one of the memory-management schemes by which a computer can store and retrieve data from secondary storage for use in main memory.'


                      Basically, if you exceed the physical memory, the job will just take forever because a lot of instructions will be only to swap pages between physical memory and swap memory.

                      Agreed, but this is something Ray did, I do not, to my knowledge have a way to specify how much swap to use. What can I do to prevent this?
                      Ray, presumably, requested memory, and page faults occured because physical memory was exausted.

                      You can't specify not to use swap because the programmer, and the program executing are unaware of which addresses are resident in physical memory and which are not.


                      Luckily, it seems you can limit the physical and virtual memory available to any process using a properly configured PBS scheduler: http://wiki.hpc.ufl.edu/index.php/PBS_Directives#Memory



                      You use PBS to launch the job.

                      According to the post numbered 69, your script is the following:
                      Discussion of next-gen sequencing related bioinformatics: resources, algorithms, open source efforts, etc


                      Code:
                      ###Parameterized PBS Script ####
                      #PBS -S /bin/bash
                      #PBS -N LUNDE.Ray4
                      #PBS -l nodes=256
                      #PBS -l walltime=25:00:00
                      #PBS -q normal
                      #PBS -j oe
                      #PBS -o LUNDE.Ray4.o
                      #PBS -M ---redacted---
                      #PBS -m abe
                      
                      wd=/scratch/myfiles/LUNDE_ASSEMBLE/
                      cd $wd
                      
                      use intel-openmpi-1.4.2
                      use Ray-1.2.0
                      
                      mpirun -np 256 Ray -p $wd\Lunde_1.fastq $wd\Lunde_2.fastq -o Lunde-contigs
                      To properly understand the parameters of the PBS scheduler, I have read http://wiki.hpc.ufl.edu/index.php/PBS_Directives
                      Your setup seems OK to me although I never worked with PBS.



                      In the post numbered 61 http://seqanswers.com/forums/showpos...8&postcount=61,

                      you wrote this:

                      Req[0] TaskCount: 256 Partition: anon
                      Utilized Resources Per Task: PROCS: 120.18 MEM: 2596M SWAP: 881G
                      Avg Util Resources Per Task: PROCS: 120.18
                      Max Util Resources Per Task: PROCS: 237.11 MEM: 2596M SWAP: 881G
                      Average Utilized Memory: 1641.30 MB
                      Average Utilized Procs: 48473.59
                      NodeSet=ONEOF:FEATURE:awesometown
                      NodeAccess: SINGLEJOB
                      NodeCount: 32

                      As I understand it, you have 256 tasks, and a task utilizes 120 processors, 2596 MiB of physical memory, and 881 GiB of swap space.

                      For all that matters, the node set is named 'awesometown'.

                      That does not make sense.

                      Too, 'Average Utilized Procs: 48473.59' does not make sense neither.



                      Your PBS script might be wrongly written or PBS might be misconfigured.

                      To ascertain the wholly truth, get support from your compute department.



                      As I promised in the post numbered 66 http://seqanswers.com/forums/showpos...4&postcount=66, I tested Ray with the Intel compiler.

                      I compiled Ray 1.2.1 with Open-MPI 1.4.3 and the Intel compiler, version 11.1.059.

                      I then launched a job, using mpirun (from Open-MPI 1.4.3) compiled with the Intel compiler. The script for Sun Grid Engine follows.

                      Code:
                      #!/bin/bash
                      #$ -N Ray1.2.1
                      #$ -P nne-790-aa
                      #$ -l h_rt=0:20:00
                      #$ -pe node 64
                      #$ -M sebastien.boisvert.3@<removed>
                      #$ -R y
                      #$ -m bea
                      module load compilers/intel/11.1.059 mpi/openmpi/1.4.3_intel
                      /software/MPI/openmpi-1.4.3_gcc/bin/mpirun /home/sboisver12/Ray/trunk/code/Ray  \
                      -p /home/sboisver12/nne-790-aa/SRA001125/SRR001665_1.fastq /home/sboisver12/nne-790-aa/SRA001125/SRR001665_2.fastq \
                      -p /home/sboisver12/nne-790-aa/SRA001125/SRR001666_1.fastq /home/sboisver12/nne-790-aa/SRA001125/SRR001666_2.fastq \
                      -o  Intel

                      The output in stdout is http://pastebin.com/yprcXwDe

                      Each MPI rank needed, on average, 284 MiB. The utilized distributed memory was 17 GiB.

                      I then used MUMmer (via the script named print-latex.sh in the scripts directory of the Ray distribution) to assess the quality of the assembly.

                      Code:
                      [sboisver12@colosse1 ~]$ print-latex.sh nne-790-aa/nuccore/Ecoli-k12-mg1655.fasta Intel.fasta  Ray-1.2.1
                              %  & numberOfContigs & bases & meanSize  & n50  & max   & coverage   & misassembled & mismatches & indels
                       Ray-1.2.1 & 123 & 4616336 & 37531 & 72499 &  176360 &  0.9819 & 0 & 2 & 4 \\
                      Everything's OK with the Intel compiler.


                      Cheers.

                      -seb

                      Comment


                      • #71
                        Ray 1.2.1 'stringray'

                        Dear all,

                        Ray 1.2.1 is now available.

                        Source:



                        This version fixes 2 critical flaws, can assemble polymorphic positions by
                        forcing bubble traversal, and adds an experimental feature: memory usage
                        reduction during the construction of the distributed graph.

                        A more detailed catalog of changes follows.

                        • SplayTreeIterator now iterates in preorder instead of inorder.
                        • Now the SplayTreeIterator is utilized to iterate over the vertices nstead ofstoring them in an array (takes too much memory).
                        • The forest of splay trees now contains 16384 trees instead of 4096. otethat as usual, each MPI rank has its own forest. Furthermore, the forest freezes when vertices distribution is properly finished -- which means no more splaying in the splay trees is to occur. The process ensures that vertices with low redundancy remain at leaves.
                        • Bubble traversal ensures no misassemblies as polymorphic positions (substitution, and indels) are assembled !!!
                        • Works with 454 data too (454 homopolymers error are interpreted as polymorphic positions).
                        • Added numeric indicator for files. (example: [1/9])
                        • Corrected a bug in library lengths messaging that leaded to hanging and/or Bus Error.
                        • Fixed a segmentation fault when the -a (output AMOS) is provided. Thanks to Daniel Brami from J. Craig Venter Institute (La Jolla, CA) for the timely report.
                        • Preliminary version of an algorithm to preemptively reduce the memory usage while building the distributed graph.
                        • Under GNU/Linux platforms, Ray outputs the virtual memory (VmData from /proc, that is the heap) utilized before exiting.



                        Thank you.

                        ps.

                        I am currently working on two human genome data sets: SRA000271
                        (African) and SRA010766 (Jay Flatley).

                        Comment


                        • #72
                          dnGASP

                          Seb,

                          If you haven't seen the posting already, I'd like to bring to your attention the de novo Genome Assembly Assessment Project (dnGASP). All details can be found at cnag.bsc.es, but briefly it's a project that solicits submissions of assemblies of a synthetic 1.8Gb diploid genome, which will be followed up with a workshop in April in Barcelona. Please take a look and see if you may be interested. We at the CNAG are looking at Ray as an option for assembling human genomes and other genomes that we have already sequenced here (but have been unable to assemble thus far). The project and associated workshop may be a good opportunity to gain exposure and to participate in a forum where issues regarding assembly and sequence data can be discussed. If you are interested, please sign up for the dngasp mailing list and register your team (Ray) on the cnag.bsc.es site as soon as possible in order to download the reads and submit assemblies.

                          Regards,
                          Tyler Alioto

                          Comment


                          • #73
                            Sounds interesting !

                            Comment


                            • #74
                              bug in Ray1.2.1?

                              Hey,

                              I have uploaded the last version of Ray to can obtain an amos output format. It ran during ten hours but I obtain errors as:
                              MPI_Isend(145): MPI_Isend(buf=0x2b196b013ad0, count=1, MPI_UNSIGNED_LONG_LONG, dest=23612, tag=89, MPI_COMM_WORLD, request=0x7fff25927d24) failed
                              MPI_Isend(95).: Invalid rank has value 23612 but must be nonnegative and less than 16

                              Do you konw this problem (and have you a solution)?

                              Best regards

                              PS: on the same data with a default output format, it was ok...!

                              Comment


                              • #75
                                Sanger reads

                                Hi seb567,

                                I have two questions.

                                1) Can Ray handle sanger reads such as WGS, Fosmid end ?

                                2) Does Ray do scaffolding ? The results did not contain N even though
                                I used paired end as following.

                                Ray \
                                -s \
                                solexa/xxx.2.2.41.single.fasta \
                                -p \
                                solexa/xxx.2.2.41.1.fasta \
                                solexa/xxx.2.2.41.2.fasta \
                                -o \
                                ray/ray.kmer31.contig \
                                -k \
                                31
                                The header line of fasta file is modified by SOAPdenovo correction tool.
                                Is this problem ?

                                Thanks,
                                Corthay

                                Comment

                                Latest Articles

                                Collapse

                                • seqadmin
                                  Exploring the Dynamics of the Tumor Microenvironment
                                  by seqadmin




                                  The complexity of cancer is clearly demonstrated in the diverse ecosystem of the tumor microenvironment (TME). The TME is made up of numerous cell types and its development begins with the changes that happen during oncogenesis. “Genomic mutations, copy number changes, epigenetic alterations, and alternative gene expression occur to varying degrees within the affected tumor cells,” explained Andrea O’Hara, Ph.D., Strategic Technical Specialist at Azenta. “As...
                                  07-08-2024, 03:19 PM
                                • seqadmin
                                  Exploring Human Diversity Through Large-Scale Omics
                                  by seqadmin


                                  In 2003, researchers from the Human Genome Project (HGP) announced the most comprehensive genome to date1. Although the genome wasn’t fully completed until nearly 20 years later2, numerous large-scale projects, such as the International HapMap Project and 1000 Genomes Project, continued the HGP's work, capturing extensive variation and genomic diversity within humans. Recently, newer initiatives have significantly increased in scale and expanded beyond genomics, offering a more detailed...
                                  06-25-2024, 06:43 AM

                                ad_right_rmr

                                Collapse

                                News

                                Collapse

                                Topics Statistics Last Post
                                Started by seqadmin, Yesterday, 07:20 AM
                                0 responses
                                20 views
                                0 likes
                                Last Post seqadmin  
                                Started by seqadmin, 07-16-2024, 05:49 AM
                                0 responses
                                36 views
                                0 likes
                                Last Post seqadmin  
                                Started by seqadmin, 07-15-2024, 06:53 AM
                                0 responses
                                40 views
                                0 likes
                                Last Post seqadmin  
                                Started by seqadmin, 07-10-2024, 07:30 AM
                                0 responses
                                41 views
                                0 likes
                                Last Post seqadmin  
                                Working...
                                X