K O N R A D
H I N S E N
We redo
Science !
ReScience
Reproducible science is good. Replicated science is better.
N I C O L A S
R O U G I E R
Once upon a time there was a model…
(Guthrie et al., 2013)
Nice paper, good results, but…
•
•
•
•
•
•
•
No public repository
No easy access
No version control
Model description included ambiguous information
Sources were mixing actual computation and GUI code
Model was split into a hundred files
Main file was 6,000 lines long
Model was hardly reproducible.
R E S C I E N C E . G I T H U B . I O
Did you say reproducible science ?
(Crook et al., 2011)
“Reproducible experiments are the cornerstone of science:
only observations that are independently confirmed enter
the body of scientific knowledge.
Computational science should excel in reproducibility, as
simulations on digital computers avoid many of the small
variations that are beyond the control of the experimental
biologist or physicist.”
Reality is different, though.
xkcd.com
R E S C I E N C E . G I T H U B . I O
Three months of hard work for a 200 lines script...
(Topalidou et al., 2015)
Three months of hard work:
• Use of the Python language and numerical libraries
• Use of the DANA library for intuitive description
• Use of the IPython notebook for interactive sessions
Source is now a single file of ~200 readable lines.
Without this replication effort, model would have been dead
and forgotten by the community.
Bill Waterson
R E S C I E N C E . G I T H U B . I O
What is replication ?
Bob reads Alice’s paper, takes note of all model properties
and then implements the model himself using a method of his
choice.
Bob confirms Alice’s result by obtaining qualitatively the
same results.
Alice’s model is declared reproducible.
octodex.github.com
R E S C I E N C E . G I T H U B . I O
Did you say replication ?
During the course of a PhD, it is often the case that a student
will try to replicate results from the literature, possibly
interacting with the original authors.
Such replication generally lives inside the hard-drive of the
computer’s student while it would be actually useful for the
whole scientific community.
What do we do then ? Let’s review it & publish it !
octodex.github.com
R E S C I E N C E . G I T H U B . I O
Can we make such effort sustainable ?
Yes we can !™
ReScience is a peer-reviewed journal that target any
computational research and encourage the explicit
reproduction of already published research promoting new
and open-source implementations.
ReScience lives on github where each new implementation is
made available together with explanations (article)
And of course, we privilege Python replications…
R E S C I E N C E . G I T H U B . I O
Why git & github ? free & popular
GitHub offers a web-based git repository hosting service with
great specific features (issue, pull request, etc).
→ Version control
→ Public repositories
→ Transparency and verifiability
→ Easy exploration of new ideas
A kind of modern lab notebook for the computer scientist.
R E S C I E N C E . G I T H U B . I O
How to submit ?
1. Fork the ReScience-Submission repository
2. Write your code, data and article
3. Submit a pull request
4. Address reviewer’s comments and questions
octodex.github.com
5. Get published and get a DOI
R E S C I E N C E . G I T H U B . I O
Actual submission
(Topalidou and Rougier, 2015)
Authors: Meropi Topalidou & Nicolas Rougier
Editor: Tiziano Zito
Reviewers: Mehdi Khamassi & Benoît Girard
R E S C I E N C E . G I T H U B . I O
ReScience
ReScience
(_2) AMi2`+iBQM #2ir22M +Q;MBiBp2 M/ KQiQ`
+Q`iB+Q@#bH ;M;HB HQQTb /m`BM; /2+BbBQM KFBM;, +QKTmiiBQMH bim/v
PM2 MQi#H2 KQ/B}+iBQM BM Qm` BKTH2K2MiiBQM Bb i?2 `2BM7Q`+2K2Mi H2`MBM; `mH2
i?i ?b #22M ;`2iHv bBKTHB}2/X P`B;BMH mi?Q`b ?p2 #22M mbBM; [mBi2 +QKTH2t
H;Q`Bi?K 7Q` 2Mbm`BM; i?i dz+Q`iB+Qbi`BiH r2B;?ib `2 #QmM/2/ #v bB;KQB/H i`Mb72`
7mM+iBQM iQ `2T`2b2Mi T?vbB+H +QMbi`BMib QM bvMTiB+ ;`Qri? rBi? M #bQHmi2 KtBKmK
Q7 yXd8 M/ M #bQHmi2 KBMBKmK Q7 yXk8XǴX h?Bb H;Q`Bi?K Bb MQi /2b+`B#2/ BM i?2
`iB+H2- #mi 7`QK bQm`+2b- Bi TT2`b i?i Bi Bb #b2/ QM i?2 2biBKiBQM Q7 i?2 r2B;?i
;`/B2Mi HQM; i?2 bB;KQB/X q2 mb2 BMbi2/ M PD@HBF2 `mH2 ;Bp2M BM i?2 avMTb2 i#H2X
J2`QTB hQTHB/QmR- k- j M/ LB+QHb SX _Qm;B2`R- k- j
q2 T`QpB/2 #2HQr i?2 7Q`KH /2b+`BTiBQM Q7 i?2 KQ/2H ++Q`/BM; iQ i?2 T`QTQbBiBQM
Q7 LQ`/HB2 2i HX (k) 7Q` `2T`Q/m+B#H2 /2b+`BTiBQMb Q7 M2m`QMH M2irQ`F KQ/2HbX
R AL_A "Q`/2mt am/@Pm2bi- "Q`/2mt- 6`M+2X k G"_A- lMBp2`bBiû /2 "Q`/2mt- AMbiBimi
SQHvi2+?MB[m2 /2 "Q`/2mt- *2Mi`2 LiBQMH /2 H _2+?2`+?2 a+B2MiB}[m2- lJ_ 83yy- hH2M+26`M+2X j AMbiBimi /2b JH/B2b L2m`Q/û;ûMû`iBp2b- lMBp2`bBiû /2 "Q`/2mt- *2Mi`2 LiBQMH /2
H _2+?2`+?2 a+B2MiB}[m2- lJ_ 8kNj- "Q`/2mt- 6`M+2X
h#H2
.2b+`BTiBQM
LB+QHbX_Qm;B2`!BM`BX7`
SQTmHiBQMb
*Q`i2t UKQiQ`- bbQ+BiBp2 +Q;MBiBp2Vai`BimK UKQiQ`- bbQ+BiBp2 +Q;MBiBp2V:SB UKQiQ` +Q;MBiBp2VahL UKQiQ` +Q;MBiBp2Vh?HKmb UKQiQ` +Q;MBiBp2V
Ĝ
PM2 iQ QM2- QM2 iQ KMv U/Bp2`;2MiV- KMv iQ QM2 U+QMp2`;2MiV
.vMKB+ `i2 KQ/2H
Ĝ
GBM2` bvMTb2
_2BM7Q`+2K2Mi H2`MBM; `mH2
1ti2`MH +m``2Mi BM +Q`iB+H `2b UKQiQ`- bbQ+BiBp2 +Q;MBiBp2V
6B`BM; `i2 T2`7Q`KM+2b
1/BiQ`
hBxBMQ wBiQ
`272`2M+2 BKTH2K2MiiBQM Q7
_2pB2r2`b
"2MQŗi :B``/
J2?/B E?KbbB
_2+2Bp2/ CmM N- kyR8
++2Ti2/ m; Rk- kyR8
Sm#HBb?2/ m; R9- kyR8
GB+2M+2 **@"u@a
*QKT2iBM; AMi2`2bib,
h?2 mi?Q`b ?p2 /2+H`2/ i?i
MQ +QKT2iBM; BMi2`2bib 2tBbiX
ă *Q/2 `2TQbBiQ`v
h#H2 R, JQ/2H /2b+`BTiBQM 7QHHQrBM; (k) T`2b+`BTiBQMX
hQTQHQ;v
*QMM2+iBpBiv
L2m`QM KQ/2H
*?MM2H KQ/2H
avMTb2 KQ/2H
SHbiB+Biv
AMTmi
_2+Q`/BM;b
¹ AMi2`+iBQM #2ir22M +Q;MBiBp2 M/ KQiQ` +Q`iB+Q@#bH ;M;HB HQQTb /m`BM;
/2+BbBQM KFBM;, +QKTmiiBQMH bim/v- JX :mi?`B2- X G2#HQBb- X :`2MM2- M/
hX "Q`m/- CQm`MH Q7 L2m`QT?vbBQHQ;v- RyN- kyRjX
AMi`Q/m+iBQM
q2 T`QTQb2 `272`2M+2 BKTH2K2MiiBQM Q7 (R) i?i BMi`Q/m+2b M +iBQM b2H2+iBQM K2+?@
MBbK BM +Q`iB+Q@#bH ;M;HB HQQTb #b2/ QM +QKT2iBiBQM #2ir22M i?2 TQbBiBp2 722/@
#+F- /B`2+i Ti?rv i?`Qm;? i?2 bi`BimK M/ i?2 M2;iBp2 722/#+F- ?vT2`/B`2+i
Ti?rv i?`Qm;? i?2 bm#i?HKB+ Mm+H2mbX h?2 Q`B;BMH BKTH2K2MiiBQM rb K/2 BM
.2HT?B UP#D2+i Sb+HV r?Qb2 bQm`+2b `2 pBH#H2 QM `2[m2bi iQ Mv Q7 i?2 mi?Q`
Q7 i?2 Q`B;BMH `iB+H2X q2 ?p2 mb2/ i?2b2 bQm`+2b iQ /BbK#B;mi2 K#B;mQmb M/
KBbbBM; BM7Q`KiBQM BM i?2 Q`B;BMH `iB+H2X h?2 `272`2M+2 BKTH2K2MiiBQM r2 T`QTQb2
?b #22M +Q/2/ BM Svi?QM 7Q` 2b2 Q7 `2/BM; M/ *vi?QM 7Q` T2`7Q`KM+2b #2+mb2
i?2 KBM `2bmHi BM+Hm/2b #i+? Q7 k8y 2tT2`BK2Mib Qp2` Rky i`BHb i?i rQmH/ #2 iQQ
bHQr 7Q` `2;mH` Svi?QM b+`BTibX
J2i?Q/b
q2 mb2/ i?2 /2b+`BTiBQM Q7 i?2 KQ/2H BM i?2 Q`B;BMH `iB+H2 b r2HH b i?2 bQm`+2b Q7
i?2 KQ/2H U`2[m2bi2/ 7`QK mi?Q`V i?i `2 K/2 Q7 ?mM/`2/ }H2b M/ e-yyy HBM2b
Q7 .2HT?B 7Q` i?2 KBM bQm`+2X q2 ?p2 #22M mM#H2 iQ +QKTBH2 i?Bb Q`B;BMH BKTH2@
K2MiiBQM #mi r2 r2`2 #H2 iQ `mM i?2 T`QpB/2/ qBM/Qrb 2t2+mi#H2X q2 7QmM/ bQK2
7+imH 2``Q`b BM i?2 Q`B;BMH `iB+H2 i?i ?p2 #22M +Q``2+i2/ BM i?Bb BKTH2K2MiiBQMX
h?2 BMBiBHBxiBQM Q7 r2B;?ib `2 /2}M2/ BM irQ /Bz2`2Mi T`ib Q7 i?2 TT2`X 6B`bi QM
T;2 jyjy Ub2+QM/ +QHmKMV dzq2B;?ib r2`2 BMBiBHBx2/ iQ :mbbBM /Bbi`B#miBQM rBi? K2M Q7 yX8 M/ a. Q7 yXyy8 i i?2 bi`i Q7 2+? bBKmHiBQMĘǴ- i?2M QM T;2 jyjR BM
i?2 +TiBQM Q7 };m`2 9- dzHH bvMTiB+ r2B;?ib r2`2 BMBiBHBx2/ iQ yX8ǴX Ai ?TT2M2/ i?i
#Qi? /2}MBiBQMb `2 `B;?i #mi /Q MQi //`2bb i?2 bK2 T`QD2+iBQMbX *Q`iB+Q@bi`BiH
bvMTiB+ r2B;?ib mb2 :mbbBM /Bbi`B#miBQM r?BH2 HH Qi?2` r2B;?ib `2 b2i iQ yX8X 6m`@
i?2`KQ`2- i?2 "QHixKMM 2[miBQM ;Bp2M BM i?2 Q`B;BMH TT2` mb2b X BMbi2/ Q7 Y
#2ir22M }`bi i2`K M/ b2+QM/ i2`KX
_2a+B2M+2 | `2b+B2M+2X;Bi?m#XBQ
R
m; kyR8 | oQHmK2 R | Abbm2 R
h#H2 k, SQTmHiBQMb
LK2
1H2K2Mib
aBx2
h?`2b?QH/
LQBb2
AMBiBH bii2
τ
*Q`i2t KQiQ`
*Q`i2t +Q;MBiBp2
*Q`i2t bbQ+BiBp2
ai`BimK KQiQ`
ai`BimK +Q;MBiBp2
ai`BimK bbQ+BiBp2
:SB KQiQ`
:SB +Q;MBiBp2
ahL KQiQ`
ahL +Q;MBiBp2
h?HKmb KQiQ`
h?HKmb +Q;MBiBp2
oHm2b UVi V
GBM2` M2m`QM
GBM2` M2m`QM
GBM2` M2m`QM
aB;KQB/H M2m`QM
aB;KQB/H M2m`QM
aB;KQB/H M2m`QM
GBM2` M2m`QM
GBM2` M2m`QM
GBM2` M2m`QM
GBM2` M2m`QM
GBM2` M2m`QM
GBM2` M2m`QM
a+H`
1×4
4×1
4×4
1×4
4×1
4×4
1×4
4×1
1×4
4×1
1×4
4×1
4
@j
@j
@j
y
y
y
YRy
YRy
@Ry
@Ry
@9y
@9y
Ĝ
RXyW
RXyW
RXyW
yXRW
yXRW
yXRW
jXyW
jXyW
yXRW
yXRW
yXRW
yXRW
Ĝ
yXy
yXy
yXy
yXy
yXy
yXy
yXy
yXy
yXy
yXy
yXy
yXy
yX8
Ry
Ry
Ry
Ry
Ry
Ry
Ry
Ry
Ry
Ry
Ry
Ry
@
h#H2 j, *QMM2+iBpBiv
aQm`+2
h`;2i
Sii2`M
q2B;?i
:BM
SHbiB+
*Q`i2t KQiQ`
*Q`i2t +Q;MBiBp2
h?HKmb KQiQ`
h?HKmb +Q;MBiBp2
(1, i) → (1, i)
(i, 1) → (i, 1)
RXy
RXy
yX9
yX9
LQ
LQ
_2a+B2M+2 | `2b+B2M+2X;Bi?m#XBQ
R E S C I E N C E . G I T H U B . I O
k
m; kyR8 | oQHmK2 R | Abbm2 R
Grand opening on September 1st 2015
Spread the word !
→ star the project on github (ReScience/ReScience)
→ follow the journal (@ReScienceEds) on twitter
→ tell your colleagues (mailing lists, blog, twitter, …)
Help the project !
octodex.github.com
→ submit articles
→ become a reviewer (contact me nicolas.rougier@inria.fr)
R E S C I E N C E . G I T H U B . I O