The Data Science and Technology (DST) Department - a part of the Computational Research Division (CRD) at Berkeley Lab - delivers leading-edge, innovative methods for solving data-intensive science problems. DST activities range from basic and applied research to deployment of software tools. Our projects span a diverse set of activities, including: data management; data movement; statistical, topological, and geometric analysis/analytics; computer vision; visualization; user-interface design; usability; end-to-end data-intensive system architecture and deployment. We focus on conceiving, developing, and applying leading-edge, innovative methods for solving data-intensive science problems. Our multidisciplinary teams are engaged on projects in five primary mission areas:
- Scientific workflows and data analysis algorithms and frameworks
- Data synthesis, management, movement, and curation of large and complex datasets
- User-centered design of interfaces and software
- Exascale data analytics and visualization capabilities
December 17, 2014
IPython Featured in Nature News
IPython was recently featured in Nature News. The IPython team is led by Fernando Perez, who joined DST earlier this month. IPython notebook makes data analysis easier to record, understand and reproduce. The article is available here and a live, interactive demo is available here
November 12, 2014
CRD Reorganization creates Data Science and Technology Department
The Data Science and Technology Department was announced today. This department brings together groups working across the span of data science problems. The new department is made up of the iintegrated Data Frameworks group led by Dan Gunter, the Scientific Data Management group led by John Wu, the Data Analytics and Visualization group led by Wes Bethel, the Usable Software systems group led by Lavanya Ramakrishnan. These groups have a long history of research and development in data science. This reorganization better aligns the groups to work together to address problems holistically. The Computational Science Department, formed at the same time, is composed of the groups focused more on a particular science and includes Craig Tull's Science Software Systems group. More details about the reorganization can be found here.
October 24, 2014
Inspiring Women in Computing
Women from Berkeley Lab's Computing Sciences area and DST delivered talks, volunteered as mentors and helped organize and energize this year's Grace Hopper Celebration of Women in Computing. More details available here.
October 7, 2014
SPOT Suite noted in NPR's Science Friday
NPR's Science Friday notes NERSC, ESnet, CRD, SLAC collaboration that uses the SPOT Suite developed by DST researchers. This was based on observations of interesting science that was submitted by listeners. More details available here.
August 18, 2014
Craig Tull Leading Multi-Disciplinary Team Enabling the SPOT Suite Transformation at ALS Beamlines
Earlier this year, the ALS became the first and only facility worldwide to fully automate GISAXS/GIWAXS measurements. This tool is primarily used to characterize the assembly and shape of nanoscopic objects at surfaces or buried interfaces in thin films—including materials like organic photovoltaics, fuel cell membranes or batteries. Combine this capability with SPOT Suite, and researchers can run experiments at this beamline from anywhere in the world, provided they have Internet access. Users can mount their samples onto barcoded sample holders at home and ship them to the ALS. At the facility, a robot arm transfers each new sample to the measurement stage, where it is automatically aligned into grazing incidence using the X-ray beam. A barcode reader informs the computer system which sample is mounted and how to run the sample. Data acquisition software then moves the sample to all angles pre-specified by the researcher and chooses the appropriate exposure time automatically for each image. As images are collected, SPOT Suite sends the data to NERSC via ESnet for scientists to access and view any time. “This automated system represents a significant leap forward in terms of labor saving, ease of use and throughput,” says Alexander Hexemer, who manages the GISAXS/GIWAXS beamline at the ALS. Read more
July 18, 2014
Deb Agarwal Among Researchers Named Senior Fellows of the Berkeley Institute for Data Science
The Berkeley Institute for Data Science has introduced its senior fellows, including Lab researchers Deborah Agarwal, E. Wes Bethel, Peter Nugent, Saul Perlmutter, David Schlegel, James Sethian, Kimmen Sjolander, Kyle Barbary, Beth Reid, and David Culler. The funded science fellows were also named with Daniela Ushizima receiving the only award to an LBL researcher (congratulations Dani!). Go here to view the complete list.
June 20, 2014
Berkeley Lab Hosts Week long Discovery Workshop covering Big Data analytics
Today, the tools available to the scientific community are undergoing a major revolution with a wide range of innovations that are enabling powerful new capabilities for knowledge discovery and analytics. The majority of these innovations are being motivated by major large scale science research initiatives. The foundation for these innovations consists of multiple new information technology architectures, tools, techniques and platforms. Berkeley Lab's Computational Research Division (CRD) and Univ. of California's AMPLab hosted a weeklong workshop on big data analytics from June 2-6.
This year, the workshop covered a wide range of topics including machine learning, graph processing, data security, tools for big data analytics. "The workshop was very successful and allowed the participants to both understand the general landscape but also dive-in deep with hands-on tutorials. The key to success of the workshop was the expertise at Berkeley Lab in these areas," said organizer Lavanya Ramakrishnan of CRD's Advanced Computing for Science Department.
In addition to Berkeley Lab and UC Berkeley, the speakers were from Cloudera, Apple, Cisco, Yahoo!, UC Davis, Nebula Inc, Microsoft, Google, HP Labs, Adatao, Databricks, Hortonworks and Amazon.
April 4, 2014
Eugen Feller Completes Inria Postdoc Visit
Eugen Feller has spent the last year visiting LBNL as an Inria@SiliconValley Post-doc as part of the DST Department/Inria Associated Team DALHIS. During his time at LBNL Eugen contributed to several research projects including Frieda and AmeriFlux. To read more about Eugen's experiences, see the interview he did once back at Inria.
December 20, 2013
Taghrid Samak Highlighted in Scientific Computing
Taghrid Samak of Berkeley Lab's Computational Research Division admits with a laugh that she wasn't one of those kids who started programming on the home computer at age 10. And if she hadn't followed her father's advice, she might have ended up looking for political solutions to pressing problems, rather than working on computational approaches to scientific challenges. Read more.
December 13, 2013
DST Intern Amy Nesky Presents Poster at the SULI Poster Session
DST intern Amy Nesky presented a poster on the work she did for the Baryon Oscillation Spectroscopic Survery (BOSS) at the Fall SULI poster session. Amy's work involved designing and developing an interactive, visual analytics website to help track and analyze the progress of the survey. See the poster for more details.
November 26, 2013
DST Team Helping Usher in a New Era of Light Source Computational Science
DST's Craig Tull was recently featured in an article for DEIXIS Magazine. The project is an LDRD involving CRD, NERSC, ALS, and ESnet personnel. DST personnel involved in the project include Craig Tull (project lead), Abdelilah Essiari, and Lavanya Ramakrishnan.
DST Team Helping Materials Science
LBNL's Kristin Persson recently co-authored an article for Scientific American about the Materials Project. The project here at LBNL involves personnel from many divisions across the lab. DST personnel involved in the project include Dan Gunter and Miriam Brafman.
October 22, 2013
TechWomen Participants visit LBNL
On October 18, participants and mentors from the TechWomen program visited LBNL for a tour of its facilities, including the Advanced Light Source. After the tour, DST department head Deb Agarwal gave a presentation on how to be an exceptional leader. The visit was organized by DST's Taghrid Samak.
December 2, 2013
Craig Tull and Team Reimagining ALS Data Environment with SPOT Suite
DEIXIS Magazine Annual 2013 featured Craig Tull, DST, Dula Parkinson, ALS, Jack Deslippe, NERSC, and others in an article about the growing flow of data from light sources. The team is working to move beamline data in real time via ESnet to some of the nation's most powerful open-science computers at NERSC, where it is processed, analyzed and visualized on the fly. Tull and team are working with the Advanced Light Source (ALS) at the Lab, but it can be extended to work with other light sources. "We're trying to move the typical data-intensive beamline into a world where they can take advantage of leadership-class, high-performance computing abilities," says Tull, who leads the project. "In the final analysis what we're trying to do is drive a quantum leap in science productivity. Read more
October 22, 2013
Deb Agarwal Among Lab Women Honored for Contributions to Science, Education
Deborah Agarwal, head of CRD's Advanced Computing for Science Department, was among 15 women honored October 18 during the first annual Women@The Lab event. Sponsored by the lab's Diversity and Inclusion Office and the Women Scientists and Engineers Council, the event highlighted the women's contributions to science and technology as well as the lab's commitment to diversity and its support for the Science, Technology, Engineering and Mathematics (STEM) workforce.
October 11, 2013
DST members participated in the LBNL 2013 Runaround
July 30, 2013
DST Summer Students contributing on projects
As the summer starts to wind toward a new school year, we want to acknowledge the excellent work of our summer students. Here is a list of this year's summer's students and a brief title for what they worked on while they were here.
- Tonglin Li, from Illinois Institute of Technology, Chicago. FRIEDA state management in cloud environments.
- Zhao Zhang, University of Chicago. Next-generation infrastructure support for mixed workloads.
- Morgan Hargrove, Louisiana State University - Materials Project workflow interface.
- Ryan Rodriguez, University of California Santa Cruz. Tigres visual representation of workflows.
- Ahmed el Hassany, Indiana University. Interoperability between ESnet lookup service and IU's lookup/topology service (UNIS) to avoid fragmentation of perfSONAR landscape going forward (Joint with Esnet).
- Karlyn Harrod, University of St. Thomas. Energy consumption models for distributed systems.
- Jin Huang, University of Texas at Arlington. Machine learning for modeling network utilization.
July 26, 2013
Sarah Poon Organizes Talks and Tour for East Bay Consortium High School Students
Sarah Poon of DST along with Boun Khamnouane of the East Bay Consortium of Educational Institutions organized a visit of about 50 high school students from the East Bay to learn about careers in science, technology, engineering and mathematics. Read more
April 2, 2013
Taghrid Samak Works to Impact Social Development in Egypt
Since the Egyptian uprising that ultimately toppled the 30-year reign of Hosni Mubarak began on Jan. 25, 2011, Taghrid Samak of Berkeley Lab’s Computational Research Division has watched as the initial hope for her homeland has unraveled into a “messy” situation, as she puts it. But last month, Samak was at MIT, meeting with other Egyptian professionals to take concrete steps to address at least some of the pressing issues in the country that launched the Arab Spring. She chaired the 2013 EgyptNEGMA (Networking, Entrepreneurship, Growth, Mobilization, and Action) conference to review 10 finalist proposals for advancing social development in Egypt and choosing the top three. Read more
January 31, 2013
DST Team Contributes to Developing Tools to Reduce Greenhouse Gases at the Source
Despite advances in alternative energy sources, the United States will continue to rely on coal-fired power plants to generate much of the nation's electricity for the next 20 years or more. While coal is an economically viable fuel, its environmental cost is high-in 2011, coal accounted for 34 percent of the energy-related carbon dioxide (CO2) emissions in the United States. This is why a U.S. Department of Energy project, called the Carbon Capture Simulation Initiative (CCSI), is bringing together America's national laboratories, industry and academic institutions, to develop and deploy state-of-the-art computational modeling and simulation tools to accelerate the commercialization of carbon capture technologies in power plants. As part of this collaboration, computational researchers at Lawrence Berkeley National Laboratory (Berkeley Lab) are playing key roles in the development of the computational tools. on its industry advisory board. Read more
In the context of CCSI, Joshua Boverhof of CRD's Advanced Computing for Science Department developed the Turbine Science Gateway (TSG), a code for running the AspenTech process simulation applications in parallel on cloud computing systems, clusters, or on standalone machines. He recently won honorable mention for TSG in a competition sponsored by Amazon. Read more