News and Announcements from OSG Operations > Operations Service Update - Tuesday, December 12 at 13:00 UTC

The GOC will upgrade the following services beginning Tuesday, December 12 at 13:00 UTC. The GOC reserves 8 hours in the unlikely event unexpected problems are encountered.

OASIS
  * Update cvmfs software on oasis and oasis-replica to the latest versions.
  * Rebuild oasis.opensciencegrid.org cvmfs repo from staged files in order to clean out unchunked large files.  This also removes subdirectories whose staged files had been removed: belle, gm2, lbne, icecube, and uc3.
  * Reinstall oasis-login from scratch in order to test that the reinstall process can be done cleanly, including installing the ouser.mis cron job.
  * Set atlas-nightlies.cern.ch repo to use CERN stratum 1 instead of stratum 0.
  * Other minor improvements to scripts on oasis and oasis-replica.

All Services
  * Operating system updates, reboots will be required. The usual HA mechanisms will be used, but some services will experience brief outages.

News and Announcements from OSG Operations > Announcing OSG CA Certificate Update

We are pleased to announce a data release for the OSG Software Stack.
Data releases do not contain any software changes.

This release contains updated CA Certificates based on IGTF 1.88:
- updated UKeScience 2B ICA based on a SHA-2 family digest (UK)
- added new PKIUNAMgrid (2017) trust anchor for roll-over (MX)

Release notes and pointers to more documentation can be found at:

http://opensciencegrid.github.io/docs/release/3.4/release-3-4-5-3/
http://opensciencegrid.github.io/docs/release/3.3/release-3-3-30-3/

Need help? Let us know:

http://opensciencegrid.github.io/docs/common/help/

We welcome feedback on this release!

News and Announcements from OSG Operations > Announcing VO Package Update v76

We are pleased to announce a data release for the OSG Software Stack.
Data releases do not contain any software changes.

This release contains VO Package v76:
- Drop redundant geant4-lcgadmin objects
- Add missing SNO+ VOMS servers
- Add notice to voms-mapfile-default

Release notes and pointers to more documentation can be found at:

http://opensciencegrid.github.io/docs/release/3.4/release-3-4-5-2/
http://opensciencegrid.github.io/docs/release/3.3/release-3-3-30-2/

Need help? Let us know:

http://opensciencegrid.github.io/docs/common/help/

We welcome feedback on this release!

News and Announcements from OSG Operations > Operations Service Update - Monday, November 27th beginning 13:00 UTC

The GOC will upgrade the following services on November 27th beginning 13:00 UTC. The GOC reserves 8 hours in the unlikely event unexpected problems are encountered.

MyOSG, OIM, Ticket
  * Changing email address from goc@opensciencegrid.org to help@opensciencegrid.org

OIM
  * Changing documentation links from TWiki to OIM internal pages.
  * Changing notification text for resource and VO registration.

JIRA
  * Addressing spurious error notices for epic projects.

News and Announcements from OSG Operations > Announcing OSG Software version 3.3.30

We are pleased to announce OSG Software version 3.3.30.

Changes to OSG 3.3.30 include:
- OSG PKI: Certificate requests are authenticated by default
- BLAHP: Fixed bug in Slurm memory-use parsing that caused jobs to be held
- HTCondor: Fixed issue validating VOMS proxies
- XRootD 4.7.1: Fixed occasional crash when LCMAPS callout to GUMS fails
- CVMFS 2.4.2: Server side bug fixes
- GridFTP-HDFS: Added support for CMVFS checksums
- Globus GridFTP server: Fixed IPv6 redirection and IPv4 EPSV response
- LCMAPS VOMS Plugin: Documented how to map using all FQANs
- RSV: Fixed CRL freshness probe, removed unused probes

Note: This is the last release where the target release date is the second
Tuesday of the month. In the future, we will release whenever we have
significant or pressing changes.

Release notes and pointers to more documentation can be found at:

http://opensciencegrid.github.io/docs/release/3.3/release-3-3-30/

Need help? Let us know:

http://opensciencegrid.github.io/docs/common/help/

We welcome feedback on this release!

News and Announcements from OSG Operations > Announcing OSG Software version 3.4.5

We are pleased to announce OSG Software version 3.4.5.

Changes to OSG 3.4.5 include:
- OSG PKI: Certificate requests are authenticated by default
- BLAHP: Fixed bug in Slurm memory-use parsing that caused jobs to be held
- HTCondor: Updated to 8.6.8 and fixed issue validating VOMS proxies
- XRootD 4.7.1: Fixed occasional crash when LCMAPS callout to GUMS fails
- CVMFS 2.4.2: Server side bug fixes
- Globus GridFTP server: Fixed IPv6 redirection and IPv4 EPSV response
- LCMAPS VOMS Plugin: Documented how to map using all FQANs
- RSV: Fixed CRL freshness probe, removed unused probes
- Upcoming: Updated to HTCondor 8.7.5 and BLAHP 1.18.34

Note: This is the last release where the target release date is the second
Tuesday of the month. In the future, we will release whenever we have
significant or pressing changes.

Release notes and pointers to more documentation can be found at:

http://opensciencegrid.github.io/docs/release/3.4/release-3-4-5/

Need help? Let us know:

http://opensciencegrid.github.io/docs/common/help/

We welcome feedback on this release!

News and Announcements from OSG Operations > Operations Service Update - Tuesday, November 20th beginning 14:00 UTC

The GOC will upgrade the following services on November 20th beginning 14:00 UTC. The GOC reserves 8 hours in the unlikely event unexpected problems are encountered.

MyOSG
  * Installing nodejs and associated libraries.

Event
  * Configuration Change, memory and data usage threshold. Increasing available resources to survive 4 day outages on consumer side.

Condor Project News > HTCondor Security Release: 8.6.8 and 8.7.5 ( November 14, 2017 )

The HTCondor team is pleased to announce the release of HTCondor 8.6.8 and HTCondor 8.7.5. These releases contain an important fix for a security issue. Affected users should update as soon as possible. More details on the security issue are in the Vulnerability Report HTCondor binaries and source code are available from our Downloads page.

Derek's Blog > Cleaning Up GRACC

The GRid ACcounting Collector (GRACC) is the OSG’s new version of accounting software, replacing Gratia. It has been running in production since March 2017. Last week, on Friday November 3rd, we held a GRACC Focus Day. Our goal was to clean up data that is presented in GRACC. My changes where:

  • Update the GRACC-Collector to version 1.1.8. The primary change in this release is setting the messages sent to RabbitMQ to be “persistent”. The persistent messages are then saved to disk in order to survive a RabbitMQ reboot.
  • Use case-insenstive comparisons to determine the Open Science Grid Information Management system (OIM) information. This was an issue with GPGrid (Fermilab), which was registered as GPGRID.
  • Set the OIM_Site equal to the Host_description attribute if the OIM logic is unable to determine the registered OIM site. This is especially useful for the LIGO collaboration, which uses sites in Europe that are not registered in OIM. Now, instead of a lot of Unknown sites listed on the LIGO site listing, it shows the somewhat reported site name of where the job ran.
GRACC Projects Page GRACC Projects Page for LIGO

Regular Expression Corrections

One of the common problems we have in GRACC is poor data coming from the various probes installed at hundreds of sites. We don’t control the data coming into GRACC, so occasionally we must make corrections to the data for clarity or correctness. One of these corrections is misreporting the “site” that the jobs ran on.

In many instances, the probe is unable to determine the site and simply lists the hostname of the worker node where the job ran. This can cause the cardinality of sites listed in GRACC to increase dramatically as we get new hostnames inserted into the sites listing. If the hostnames are predictable, a regular expression matching algorithm can match a worker node hostname to a proper site name.

The largest change for GRACC was the regular expression corrections. With this new feature, GRACC administrators can set corrections to match on attributes using regular expression patterns. For example, consider the following correction configuration.

[[Corrections]]
index = 'gracc.corrections'
doc_type = 'host_description_regex'
match_fields = ['Host_description']
source_field = 'Corrected_OIM_Site'
dest_field = 'OIM_Site'
regex = true

This configuration means:

Match the Host_description field in the incoming job record with the regular expression Host_description field in the corrections table. If they are a match, take the value in the Corrected_OIM_Site field in the corrections table and place it into the OIM_Site field in the job record.

And the correction document would look like:

{
  "_index": "gracc.corrections-0",
  "_type": "host_description_regex",
  "_id": "asldkfj;alksjdf",
  "_score": 1,
  "_source": {
    "Host_description": ".*\.bridges\.psc\.edu",
    "Corrected_OIM_Site": "PSC Bridges",
  }
}

The regular expression is in the Host_description FIELD.

So, if the incoming job record is similar to :

{
...
"Host_description": "l006.pvt.bridges.psc.edu"
...
}

Then the correction would modify or create values such that the final record would approximate:

{
...
"Host_description": "l006.pvt.bridges.psc.edu",
"OIM_Site": "PSC Bridges",
"RawOIM_Site": ""
...
}

Note that the Host_description field stays the same. We must keep it the same because it is used in record duplicate detection. If we modified the field and resummarized previous records, then it would cause multiple records to represent the same job.

The GRid ACcounting Collector (GRACC) is the OSG’s new version of accounting software, replacing Gratia. It has been running in production since March 2017. Last week, on Friday November 3rd, we held a GRACC Focus Day. Our goal was to clean up data that is presented in GRACC. My changes where:

Pegasus news feed > Pegasus at SC 2017

Are you going to attend the SC17 conference in Denver, Colorado on November 12-17, 2017?

We will be giving a presentation “Software Integrity with Pegasus:Securing Scientific Workflow Data”  about our work as part of the  NSF SWIP project on

  • Tuesday, Nov 14th at 2:00pm at the University of Southern California booth #1731.
  • Wednesday, Nov 15th at 10:30am at the Indiana University booth #601.
  • Wednesday, Nov 15th at 11:30am at the RENCI booth #437.

There will be a Pegasus Introduction and Upcoming Features Talk on

  • Wednesday, Nov 15th at 2:00pm at the University of Southern California booth #1731.

We will be holding Pegasus office hours on

  • Tuesday, Nov 14th at 3:00 pm – 4:00 pm at the University of Southern California booth #1731.
  • Wednesday, Nov 15th at 3:00 pm – 4:00 pm at the University of Southern California booth  #1731.

If the above times don’t work for you and you would like to meet with us, you can reach us at pegasus aT isi dot edu to schedule a meeting at SC.

There will also be presentation Panorama 360: Performance Data Capture and Analysis for End-to-end Scientific Workflows about our work as part of DOE Panorama Project on

  • Tuesday, Nov 14th at 12:30 pm – 1:30 pm at the RENCI booth #437.
  • Thursday, Nov 16th at 11:30 am – 12:30 pm at the RENCI booth #437.

Two research papers will be presented in the WORKS workshop. Please, join us in the workshops and let’s have some coffee and very interesting discussions.

WORKS 2017: 12th Workflows in Support of Large-Scale Science (Monday, November 13, 2017):

  • S. Callaghan, G. Juve, K. Vahi, P. J. Maechling, T. H. Jordan, and E. Deelman, “rvGAHP – Push-based Job Submission using Reverse SSH Connections,” in 12th Workshop on Workflows in Support of Large-Scale Science (WORKS’17), 2017.
    [Bibtex]
    @inproceedings{callaghan-works-2017,
    title = {{rvGAHP} - Push-based Job Submission using Reverse SSH Connections},
    author = {Callaghan, Scott and Juve, Gideon and Vahi, Karan and Maechling, Philip J. and Jordan, Thomas H. and Deelman, Ewa},
    booktitle = {12th Workshop on Workflows in Support of Large-Scale Science (WORKS'17)},
    year = {2017},
    pages = {},
    doi = {},
    note = {Funding Acknowledgments: NSF 1664162 and 1443047}
    }
  • [PDF] [DOI] R. Ferreira da Silva, S. Callaghan, and E. Deelman, “On the Use of Burst Buffers for Accelerating Data-Intensive Scientific Workflows,” in 12th Workshop on Workflows in Support of Large-Scale Science (WORKS’17), 2017.
    [Bibtex]
    @inproceedings{ferreiradasilva-works-2017,
    title = {On the Use of Burst Buffers for Accelerating Data-Intensive Scientific Workflows},
    author = {Ferreira da Silva, Rafael and Callaghan, Scott and Deelman, Ewa},
    booktitle = {12th Workshop on Workflows in Support of Large-Scale Science (WORKS'17)},
    year = {2017},
    pages = {},
    doi = {10.1145/3150994.3151000},
    note = {Funding Acknowledgments: DOE DE-SC0012636, NSF 1664162}
    }

SC Floor Plan with Booth Locations highlighted can be found here.

272 views


News and Announcements from OSG Operations > Announcing changes to the OSG software release model

After the November 14 release, the software and release team will be moving to a flexible release model [1]. Under this model, we'll be free to do releases on days other than the second Tuesday of every month.

We are making this change to deliver upstream updates more quickly without having to designate them as high-priority software releases. As a user, you shouldn't notice much of a difference since we are aiming for approximately monthly software releases. We will still distribute data releases as soon as they are ready.

Let us know if you have any questions or concerns at help@opensciencegrid.org!

[1] https://opensciencegrid.github.io/technology/policy/flexible-release-model/

News and Announcements from OSG Operations > Announcing OSG CA Certificate Update

We are pleased to announce a data release for the OSG Software Stack.
Data releases do not contain any software changes.

This release contains updated CA Certificates based on IGTF 1.87:
- added new accredited classic DarkMatter Private Root G4 and ICA (AE)
- updated PK-Grid-2007 trust anchor with extended validity period (PK)
- extended validity period for UNAMgrid-ca trust anchor (MX)

Release notes and pointers to more documentation can be found at:

http://opensciencegrid.github.io/docs/release/3.4/release-3-4-4-3/
http://opensciencegrid.github.io/docs/release/3.3/release-3-3-29-3/

Need help? Let us know:

http://opensciencegrid.github.io/docs/common/help/

We welcome feedback on this release!

News and Announcements from OSG Operations > GOC Service Update, Tuesday, November 7th at 14:00 UTC

The GOC will upgrade the following services on November 7th beginning 14:00 UTC. The GOC reserves 8 hours in the unlikely event unexpected problems are encountered.

Glidein
  * Update filebeat on the GOC factory. This will not require a downtime.

Ticket
  * Security notification, CCs email notification adjustment


Condor Project News > HTCondor 8.7.4 released! ( October 31, 2017 )

The HTCondor team is pleased to announce the release of HTCondor 8.7.4. This development series release contains new features that are under development. This release contains all of the bug fixes from the 8.6.7 stable release. Enhancements in the release include: Improvements to DAGMan including support for late job materialization; Updates to condor_annex including improved status reporting; When submitting jobs, HTCondor can now warn about job requirements; Fixed a bug where remote CPU time was not recorded in the history; Improved support for OpenMPI jobs; The high availability daemon now works with IPV6 and shared_port; The HTCondor Python bindings are now available for Python 2 and 3 in pip. Further details can be found in the Development Version History and the Stable Version History. HTCondor 8.7.4 binaries and source code are available from our Downloads page.

Condor Project News > HTCondor 8.6.7 released! ( October 31, 2017 )

The HTCondor team is pleased to announce the release of HTCondor 8.6.7. A stable series release contains significant bug fixes. Highlights of this release are: Fixed a bug where memory limits might not be updated in cgroups; Add SELinux type enforcement rules to allow condor_ssh_to_job to work; Updated systemd configuration to shutdown HTCondor in an orderly fashion; The curl_plugin utility can now do HTTPS transfers; Specifying environment variables now works with the Python Submit class. More details about the fixes can be found in the Version History. HTCondor 8.6.7 binaries and source code are available from our Downloads page.

Condor Project News > NCSA calls on HTCondor partnership to process data for the Dark Energy Survey ( October 30, 2017 )

HPCWire recently published an aritcle on how HTCondor enables NCSA to take raw data from the Dark Energy camera telescope and process and disseminate the results within hours of observations occurring.


Subscribe