Tijmen2/cosmosage-v3.1

cosmosage

cosmosage is a natural-language cosmology assistant that can answer questions about cosmology.

cosmosage-v3.1 is the latest iteration in the cosmosage series. It fine-tuned from the Meta-Llama-3.1-8B base model. First, I performed continued pretraining on thousands of papers and textbooks relevant to cosmology. The next step was creating an assistant model by supervised fine-tuning on synthetically-generated question-answer pairs. In order to promote multi-turn conversations, broad instruction following and other general AI assistant capabilities, general-purpose instruction datasets were included in addition to the cosmology-specific question-answer pairs.

The cosmosage codebase is available at https://github.com/tijmen/cosmosage

A presentation on cosmosage was given on 2024-01-21 at QUP, KEK and is available at https://www.youtube.com/watch?v=azwfG2UTNEY

A paper was prepared and a preprint is available at https://arxiv.org/abs/2407.04420

Usage

cosmosage-v3.1 uses the same prompt template as Llama-3 and Llama-3.1.

Comparison to cosmosage-v3

cosmosage-v3.1 benefits from a few improvements over cosmosage-v3:

• The base model is Llama-3.1 rather than Llama-3.
• The continued pretraining dataset underwent perplexity-based cleaning, deleting the 1% of paragraphs with the highest perplexity according to the base model.
• The general-purpose instruct-tuning dataset that supplements the QA dataset is an improved rule-based combination of various popular datasets on huggingface. The data volume is >3 times greater than the instruction data used for cosmosage-v3.

Training details

cosmosage-v3.1 was trained on 12xA100 (40 GB) at the Gadi supercomputer, part of NCI, Australia. The continued pretraining cost ~1500 GPU-hours while the supervised fine tuning cost ~2300 GPU hours. A big thanks goes out to Yuan-Seng Ting for providing the computational resources.

Example output

User:

In what sense was WMAP a differential instrument? Was Planck a differential instrument, too?

Assistant:

WMAP was a differential instrument in the sense that the signal detected was a difference between the radiometer output from two horns pointing in different directions. However, Planck did not have a differential design.

---

User:

What do we know about the particle nature of dark matter? Summarize the possibilities and the state of the field.

Assistant:

Dark Matter (DM) is one of the most mysterious components in our Universe. It has been detected indirectly from its gravitational effects on visible matter, such as stars, galaxies, and galaxy clusters, but its particle nature remains elusive. Several candidates for DM particles have been proposed, including Weakly Interacting Massive Particles (WIMPs), axions, and Primordial Black Holes (PBHs).

1. WIMPs: WIMPs are hypothetical particles that interact via weak nuclear force and gravity, making them a popular candidate for DM. The WIMP miracle refers to the observation that the relic abundance of these particles naturally matches the observed dark matter density if their annihilation cross-section is approximately 3 x 10^-26 cm^3/s. However, despite extensive experimental searches, no direct detection or indirect signals from WIMPs have been found so far, leading some scientists to question if the WIMP miracle is truly a coincidence or if it needs to be reevaluated.
2. Axions: Axions are hypothetical low-mass particles that arise in certain extensions of the Standard Model of particle physics, like the Peccei-Quinn mechanism. They can also serve as viable dark matter candidates due to their extremely weak interaction with ordinary matter. Non-thermal production mechanisms for axions include the realignment mechanism, where they acquire their initial misalignment angle during phase transitions in the early universe, and the decay of topological defects like strings or domain walls.
3. Primordial Black Holes (PBHs): PBHs are compact objects formed in the early universe due to large-amplitude fluctuations in the energy density. These fluctuations could have originated from quantum fluctuations during inflation or from other processes. PBHs come in various masses, ranging from Planckian-scale micro-black holes to supermassive macroscopic black holes. The existence of PBHs as a significant component of dark matter depends on their mass and formation rate. Open mass windows exist for PBHs at different scales, such as those responsible for ultra-high-energy cosmic rays or LIGO/Virgo gravitational wave events. In conclusion, while there has been significant progress in understanding the astrophysical signatures of dark matter, the particle nature of dark matter still remains an open question. The three main contenders - WIMPs, axions, and PBHs - each have unique properties and potential connections to observable phenomena. As experiments continue to push the boundaries of sensitivity and new theoretical developments emerge, the nature of dark matter may finally be unveiled.

---

User:

Explain the ISW effect.

Assistant:

The integrated Sachs-Wolfe (ISW) effect is a phenomenon in cosmology that describes the interaction between cosmic microwave background (CMB) photons and gravitational potentials in an expanding universe. It provides valuable information about the distribution of matter and dark energy, as well as their effects on the evolution of the universe. The ISW effect occurs when CMB photons pass through regions with varying gravitational potential wells due to the presence of large-scale structures such as galaxy clusters or voids. As these photons travel through these potential wells, they gain energy from the falling into the well, and then lose energy while climbing out again. This change in energy manifests as a temperature fluctuation in the observed CMB radiation. In other words, the ISW effect is essentially a secondary anisotropy in the CMB caused by the late-time evolution of the gravitational potentials during the dark-energy-dominated era. Since it relies on both the distribution of matter and dark energy, it can be used to probe the properties of dark energy and test various models for its behavior. The ISW effect has been detected by cross-correlating CMB data from satellites like WMAP and Planck with large-scale structure tracers like galaxies, galaxy groups, and the Cosmic Infrared Background. These correlations provide evidence for the existence of dark energy and help constrain its equation-of-state parameter, which governs its behavior over time. In summary, the ISW effect is a powerful tool for understanding the role of dark energy in the expansion history of the universe and testing different theories for its nature.

---

User:

Explain why the time of matter-radiation equality affects the CMB damping tail.

Assistant:

The time of matter-radiation equality, which is when the energy density of matter equals that of radiation in the universe, plays a crucial role in determining the behavior of perturbations at late times and the suppression of power on small scales in the cosmic microwave background (CMB) anisotropies. This effect can be understood by considering the evolution of perturbations from recombination to today. At late times, after matter-radiation equality, all modes are inside the Hubble horizon. The growth factor D+(η)=a(η)/a(eq) must satisfy ˙D+/D+=1/η for η≫aeq. Therefore, we have δc(k,η)=δcr(k)[3 4+15 8ΩB ΩM(aeq a)−3] for η≫η∗, where δcr(k)≡[2k2T(k)]{Φ1+[Ψ1−Φ1 k2/a2]}. In the limit k→0, this expression becomes independent of the wave number, leading to the suppression of power on small scales in the CMB damping tail.

Citation

cosmosage can be cited using the following BibTeX entry:

@article{de_haan_cosmosage_2024,
    title = {cosmosage: {A} {Natural}-{Language} {Assistant} for {Cosmologists}},
    shorttitle = {cosmosage},
    url = {http://arxiv.org/abs/2407.04420},
    abstract = {cosmosage is a natural-language assistant intended for a wide audience, from laypersons interested in cosmology to students, teachers, and professional cosmologists. cosmosage provides a novel way to access knowledge and reason about cosmology. Leveraging the power of advanced large language models (LLMs), cosmosage has learned from a vast corpus of open-access source texts, including textbooks and papers. cosmosage is found to be state-of-the-art on the narrow task of answering questions about cosmology, outperforming all general-purpose models. The model parameters and code are publicly available.},
    urldate = {2024-07-08},
    publisher = {arXiv},
    author = {de Haan, Tijmen},
    month = jul,
    year = {2024},
    note = {arXiv:2407.04420 [astro-ph]},
    keywords = {Astrophysics - Cosmology and Nongalactic Astrophysics, Astrophysics - Instrumentation and Methods for Astrophysics},
}