The first paragraph should summarize the reading. The second paragraph should highlight an specific point and/or briefly explore something that interested you (e.g., you may wish to focus on one aspect of the paper in more depth, you may wish to discuss something in the reading that you disagree with). Each paragraph represents one point of the assignment. During the lecture, we will draw upon your reports for some group discussion.
You should submit a summary paragraph and idea highlight paragraphper each separate article.(only chapter 4 for both 2.pdf and 3.pdf
() ar X iv :1 30 1. 70 64 v1 [ as tr o- ph .I M ] 2 9 Ja n 20 13 How to Scale a Code in the Human Dimension Matthew J. Turk (
[email protected]) Columbia University This is a re-telling of a talk given at Scientific Software Days in December, 2012, at the Texas Advanced Computing Center. The slides and video of this talk can be found at http:// scisoftdays.org/ meetings/2012/ . Abstract: As scientists’ needs for computational techniques and tools grow, they cease to be supportable by software developed in isolation. In many cases, these needs are being met by communities of practice, where software is developed by domain scientists to reach pragmatic goals and satisfy distinct and enumerable scientific goals. We present techniques that have been successful in growing and engaging communities of practice, specifically in the yt and Enzo communities. 1 Why “Community?” Astrophysics, and particularly computational astro- physics, is dominated by vertically-integrated, small- population research collaborations. The concept of a community of researchers – sharing physics modules, improvements to simulation codes, analysis tech- niques, technology – is somewhat foreign. In fact, this idea of “community” is often viewed as a detriment to the individual researcher, rather than as a benefit to the field. I participate in two vibrant, active commu- nities in computational astrophysics, those surround- ing the simulation code Enzo and the analysis code yt. Within these two communities, we have found a somewhat surprising result: the empowerment of the community does not come at the expense of individual success. In fact, with actively shep- herded and cultivated community participation and processes, the opposite is true: the betterment of the community comes at the enrichment of the individ- ual. In addition to the concrete, measurable benefits we receive from community-focused development, we have also realized that community development is essential to the continued health of the field of computational astrophysics. By focusing on de- veloping a community of practice, where the goals and technology are driven by active participants, we have been able to expand the class of astrophysical prob- lems to which the technologies have been applied. This has led not to consolidation of research inter- ests, but rather a broadening, with improvements, technology, and techniques being directly shared be- tween working domain scientists. In this paper, I outline the infrastructure and the techniques with which we have cultivated these two complementary, but distinct, communities and the various conscious decisions we have made to ensure their growth and sustainability. 2 Introduction: yt and Enzo The yt Project (http://yt-project.org/) is an open source, community-developed analysis code for simulated astrophysical data. Largely written in Python, Cython and C, it is parallelized using MPI and OpenMP and has been used to analyze datasets whose size range from small (tens of megabytes) to large (tens of terabytes). yt is designed to abstract out underlying technical aspects of simulation data such as file format, units, geometric conventions, and 1 http://arxiv.org/abs/1301.7064v1 http://scisoftdays.org/meetings/2012/ http://yt-project.org/ parameter storage in such a way that is neutral to the underlying simulation platforms. The flagship simu- lation platforms, where we conducted detailed testing and support all functionality, are are Enzo, FLASH, Orion, Castro, Nyx, Piernik and NMSU-ART. In the current development branch (described briefly below) this has been expanded to include limited support for SPH and N-body codes such as Gadget, as well as full support for Octree codes such as RAMSES and ART. yt provides a language for describing physical regions and applying data processing techniques to those re- gions: rather than focusing on selecting grid patches, particles or octs and then masking out overlapping regions, it develops concepts of geometric regions, re- gions defined by fluid quantities, and processes that transform those regions into quantitative values. yt is best thought of not as an application, but as a library for asking and answering questions about data. This aspect of yt naturally encourages technical contribu- tions, as every user of the software package typically writes analysis code that builds upon its underlying machinery. Although its mission has expanded in re- cent years to include the application of microphysi- cal solvers, standardized input/output data formats, creating initial conditions, and even studying data from earthquake simulations, at its core yt remains a tool for interrogating astrophysical data and answer- ing questions about the underlying physics of that data. The defining characteristic of yt is not that it is written in Python, or the operations it does, but rather that it is supported by a participatory commu- nity of scientists interested in both using and devel- oping it. The yt community draws its members pri- marily from computational astrophysicists: individu- als who conduct and analyze simulations. The first version was written in late 2006 by and for a single scientist, but over the last several years has attracted 170 subscribers to the general mailing list and 50 for the development mailing list; in 2012 it was identi- fied as one of the most highly used codes on the NSF NICS analysis machine Nautilus [Szczepanski et al., 2012]. Over the course of its history, nearly 40 peo- ple have contributed changesets, ranging from tiny to very large, and the mailing lists are relatively active, averaging between one and four messages a day. In 2012 we held our first workshop at the FLASH center in Chicago, and we are holding a second workshop in March of 2013 at UC Santa Cruz. Despite this rela- tively high level of activity for a project in computa- tional astrophysics, the community has faced several challenges as well as taken steps to directly address these challenges. I am the original author of yt, having created it during my graduate school career to analyze and visualize data created by the simulation code Enzo (http://enzo-project.org/), an adaptive mesh re- finement simulation code. Enzo itself has undergone a number of changes over its years, both in the man- ner in which it is developed and the problems to which it can be applied. Enzo began as a code- base largely developed by a single individual, but through the stewardship of the Laboratory for Com- putational Astrophysics (LCA) it grew into a large, open source (but mostly not community-based) de- velopment project. A direct consequence of both the technology and the prevailing mindsets led to diver- gent lines of “public” and “internal” development, resulting in a highly fragmented community of users and developers and many different branches of the code itself. Over the course of a few years in 2009 and 2010, through concerted efforts from the Enzo community, it transitioned from a closed development model with periodic open releases into a fully-open, transparent development model based around con- tributions from community members. This included resolving many divergent code bases and produced the Enzo 2.0 release. The code is now developed us- ing many best practices of software development in- cluding testing, version control, peer review, infras- tructure design and investment of stakeholders in the development roadmap. This has resulted in contribu- tions of both physics modules and infrastructure im- provements, and has even brought to light bugs that have subsequently been fixed by community mem- bers. This transition, from semi-open to community- driven, has brought an energy and excitement to the development of Enzo that seems likely to sustain it for years to come. Many of the items discussed be- low, about how to shape and foster community, have been applied to the Enzo community as well as yt. 3 What is “Community”? Open Source is often used as a synonym for com- munity development, but in practice they are better thought of as two different, but overlapping, modes of development. The growth of a community, where in- dividuals participate in discussions, report problems, provide enhancements, and support other community members, is neither necessitated by software being open source, nor is it a foregone conclusion for open 2 http://enzo-project.org/ source software. For scientific software, this situation is slightly more complex, as the adoption of open source meth- ods are often met with resistance. In addition to this, members of a given community related to soft- ware are just as likely – if not more likely – to be working on similar projects, competing for mindshare among the academic public, and even competing for funding or jobs. Building communities that are able to thrive despite these barriers can result in con- siderable, non-local benefits: the aphorism “a ris- ing tide lifts all ships” is nowhere more true than in community-driven scientific codes. Even the pres- ence of other, engaged community members means that there are more people able to answer questions from newcomers and provide assistance and energy toward solving problems. The concept of openness in science amongst aca- demics is something of a paradox. Often, utilizing commodity tools is viewed as a very positive trait, whereas releasing software is occasionally viewed with skepticism. While this is changing, and in some ways quite rapidly, the idea of source code being shared be- tween potential competitors is still often seen as dan- gerous or even anathema to scientific progress. This typically breaks along three primary objections:1 1. Why should I give up my competitive advan- tage? 2. How can I manage supporting a code? 3. What if someone finds a bug in my work? Answering these questions in detail is beyond the scope of this document. But, a few comments can be made. The first two objections to releasing code are directly addressed by a community of practice. As noted above, the benefits from collab- orative, participatory communities can alleviate the burdens of support as well as provide advantages to scientific inquiry that would otherwise not be present. In both the Enzo and yt communities, as the commu- nity has scaled in size, with it has scaled the number of eager, helpful participants in contributing mod- ules (which are then shared, collectively) as well as provide support for problems and issues on the mail- ing list. When discussing these aspects of competi- tive advantage and support, it is also worthwhile to frame the discussion in terms of generalization and specialization. As an example, in the early stages of developing a simulation code, it’s entirely reasonable that a single individual can manage every aspect of the code: the IO framework, parallelism, hydrody- namics, gravity, and so on. But as the simulation code becomes larger, more general, and more mature, it becomes unwieldy for a single person to manage and develop all aspect of a code. Within Enzo we have seen this as the code has grown to include many different hydrodynamic modules (including magneto- hydrodynamics), chemistry modules, radiative trans- port, star particle implementations, and parallelism strategies. The overall generalization of the code base has resulted in a deep specialization in some areas of