[07:05:08.384] src/tools/ci/cores_backtrace.sh linux /tmp/cores [07:05:09.072] dumping /tmp/cores/postgres-3-39671.core for /tmp/cirrus-ci-build/build/tmp_install/usr/local/pgsql/bin/postgres [07:05:10.727] [New LWP 39671] [07:05:10.727] [New LWP 38424] [07:05:10.727] Downloading separate debug info for /lib/x86_64-linux-gnu/libm.so.6... [07:05:11.488] Downloading separate debug info for /lib/x86_64-linux-gnu/libdl.so.2... [07:05:12.251] Downloading separate debug info for /lib/x86_64-linux-gnu/librt.so.1... [07:05:13.467] Downloading separate debug info for /lib/x86_64-linux-gnu/libgssapi_krb5.so.2... [07:05:15.837] Downloading separate debug info for /lib/x86_64-linux-gnu/libicuuc.so.67... [07:05:15.837] Downloading separate debug info for /.cache/debuginfod_client/f871bbd529a02abd860f0d16b842b5b20234cb49/debuginfo... [07:05:17.702] Downloading separate debug info for /lib/x86_64-linux-gnu/libicui18n.so.67... [07:05:18.911] Downloading separate debug info for /lib/x86_64-linux-gnu/libldap_r-2.4.so.2... [07:05:20.479] Downloading separate debug info for /lib/x86_64-linux-gnu/libxml2.so.2... [07:05:21.543] Downloading separate debug info for /lib/x86_64-linux-gnu/liblz4.so.1... [07:05:22.306] Downloading separate debug info for /lib/x86_64-linux-gnu/libpam.so.0... [07:05:23.537] Downloading separate debug info for /lib/x86_64-linux-gnu/libssl.so.1.1... [07:05:25.150] Downloading separate debug info for /lib/x86_64-linux-gnu/libcrypto.so.1.1... [07:05:26.550] Downloading separate debug info for /lib/x86_64-linux-gnu/libsystemd.so.0... [07:05:27.459] Downloading separate debug info for /lib/x86_64-linux-gnu/libz.so.1... [07:05:28.992] Downloading separate debug info for /lib/x86_64-linux-gnu/libzstd.so.1... [07:05:30.379] Downloading separate debug info for /lib/x86_64-linux-gnu/libpthread.so.0... [07:05:30.850] [Thread debugging using libthread_db enabled] [07:05:30.850] Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1". [07:05:31.913] Downloading separate debug info for /lib64/ld-linux-x86-64.so.2... [07:05:33.299] Downloading separate debug info for /lib/x86_64-linux-gnu/libkrb5.so.3... [07:05:34.361] Downloading separate debug info for /lib/x86_64-linux-gnu/libk5crypto.so.3... [07:05:34.977] Downloading separate debug info for /lib/x86_64-linux-gnu/libcom_err.so.2... [07:05:35.742] Downloading separate debug info for /lib/x86_64-linux-gnu/libkrb5support.so.0... [07:05:38.029] Downloading separate debug info for /lib/x86_64-linux-gnu/libstdc++.so.6... [07:05:38.959] Downloading separate debug info for /lib/x86_64-linux-gnu/libgcc_s.so.1... [07:05:39.720] Downloading separate debug info for /lib/x86_64-linux-gnu/liblber-2.4.so.2... [07:05:40.630] Downloading separate debug info for /lib/x86_64-linux-gnu/libresolv.so.2... [07:05:41.544] Downloading separate debug info for /lib/x86_64-linux-gnu/libsasl2.so.2... [07:05:43.132] Downloading separate debug info for /lib/x86_64-linux-gnu/libgnutls.so.30... [07:05:44.192] Downloading separate debug info for /lib/x86_64-linux-gnu/liblzma.so.5... [07:05:44.956] Downloading separate debug info for /lib/x86_64-linux-gnu/libaudit.so.1... [07:05:46.344] Downloading separate debug info for /lib/x86_64-linux-gnu/libgcrypt.so.20... [07:05:46.960] Downloading separate debug info for /lib/x86_64-linux-gnu/libkeyutils.so.1... [07:05:48.490] Downloading separate debug info for /lib/x86_64-linux-gnu/libp11-kit.so.0... [07:05:49.253] Downloading separate debug info for /lib/x86_64-linux-gnu/libidn2.so.0... [07:05:50.478] Downloading separate debug info for /lib/x86_64-linux-gnu/libunistring.so.2... [07:05:51.246] Downloading separate debug info for /lib/x86_64-linux-gnu/libtasn1.so.6... [07:05:52.314] Downloading separate debug info for /lib/x86_64-linux-gnu/libnettle.so.8... [07:05:53.232] Downloading separate debug info for /lib/x86_64-linux-gnu/libhogweed.so.6... [07:05:54.450] Downloading separate debug info for /lib/x86_64-linux-gnu/libgmp.so.10... [07:05:55.064] Downloading separate debug info for /lib/x86_64-linux-gnu/libcap-ng.so.0... [07:05:55.983] Downloading separate debug info for /lib/x86_64-linux-gnu/libgpg-error.so.0... [07:05:56.748] Downloading separate debug info for /lib/x86_64-linux-gnu/libffi.so.7... [07:05:57.239] Core was generated by `postgres: standby2: startup waiting for 000000010000000000000006 '. [07:05:57.239] Program terminated with signal SIGQUIT, Quit. [07:05:57.239] #0 0x00007f97ab7c5fc7 in sigprocmask () from /lib/x86_64-linux-gnu/libc.so.6 [07:05:57.266] [Current thread is 1 (LWP 39671)] [07:05:57.374] [07:05:57.374] Thread 2 (Thread 0x7f97a915ca40 (LWP 38424)): [07:05:57.374] #0 0x00007f97ab888a61 in clone () from /lib/x86_64-linux-gnu/libc.so.6 [07:05:57.374] No symbol table info available. [07:05:57.374] #1 0x00007f97ab876e5e in ?? () from /lib/x86_64-linux-gnu/libc.so.6 [07:05:57.374] No symbol table info available. [07:05:57.374] #2 0x00007f97ab877497 in ?? () from /lib/x86_64-linux-gnu/libc.so.6 [07:05:57.374] No symbol table info available. [07:05:57.374] #3 0x00007f97ab876cdb in posix_spawn () from /lib/x86_64-linux-gnu/libc.so.6 [07:05:57.374] No symbol table info available. [07:05:57.374] #4 0x00007f97ab7d2a69 in ?? () from /lib/x86_64-linux-gnu/libc.so.6 [07:05:57.374] No symbol table info available. [07:05:57.374] #5 0x00005596e55f7a01 in RestoreArchivedFile (path=path@entry=0x7ffdbe155b00 "pg_wal/00000", xlogfname=xlogfname@entry=0x7ffdbe155f50 "00000001", '0' , "6", recovername=recovername@entry=0x5596e60cb5dc "RECOVERYXLOG", expectedSize=16777216, cleanupEnabled=) at ../src/backend/access/transam/xlogarchive.c:177 [07:05:57.374] xlogpath = "pg_wal/RECOVERYXLOG", '\000' , "\066", '\000' , "\b\000\000\000\000\000\000\000\334}\f\346\226U\000\000(Z\025\276\375\177\000\000\320}\f\346\226U\000\000\224 \363\345\226U\000\000P_\025\276\375\177\000\000\227\265\f\346\226U\000\000(Z\025\276\375\177\000\000\225\265\f\346\226U\000\000\030\311\025\251\227\177\000\000\360*\363\345\226U", '\000' , "\b", '\000' , "\060", '\000' , "\002", '\000' ... [07:05:57.374] xlogRestoreCmd = 0x5596e8644588 "cp \"/tmp/cirrus-ci-build/build/testrun/recovery/040_standby_failover_slots_sync/data/t_040_standby_failover_slots_sync_publisher_data/archives/00000001", '0' , "6\" \"pg_wal/RECOVERYXLOG\"" [07:05:57.374] lastRestartPointFname = "00000001", '0' , "5\000*\363\345\226U", '\000' , "\001\000\000~\232\000\000\300֕\253\227\177\000\000`ו\253\227\177\000\000~\232\000\000\002", '\000' , "\020-}\253\227\177\000\000\360R\025\276\375\177", '\000' , "W!\222\253\227\177\000\000O!\222\253\227\177\000\000\210Ed\350\226U", '\000' , "@", '\000' ... [07:05:57.374] rc = [07:05:57.374] stat_buf = {st_dev = 94106591359608, st_ino = 335544320, st_nlink = 140289393843552, st_mode = 0, st_uid = 0, st_gid = 0, __pad0 = 0, st_rdev = 13514588173772186624, st_size = 0, st_blksize = 94106632340872, st_blocks = 140727792522480, st_atim = {tv_sec = 140727792522064, tv_nsec = 140727792522192}, st_mtim = {tv_sec = 0, tv_nsec = 94106632340872}, st_ctim = {tv_sec = 140289394570391, tv_nsec = 0}, __glibc_reserved = {94106591371412, 140727792525152, 94106593033513}} [07:05:57.374] restartSegNo = [07:05:57.374] restartRedoPtr = 83886120 [07:05:57.374] restartTli = 1 [07:05:57.374] __func__ = "RestoreArchivedFile" [07:05:57.374] #6 0x00005596e5606691 in XLogFileRead (segno=140727792525056, segno@entry=6, emode=emode@entry=13, tli=tli@entry=1, source=source@entry=XLOG_FROM_ARCHIVE, notfoundOk=notfoundOk@entry=true) at ../src/backend/access/transam/xlogrecovery.c:4206 [07:05:57.374] xlogfname = "00000001", '0' , "6\000\000\000\t", '\000' , "\360\001\000\000\000\000\000\000\320c\025\276\375\177\000\000#\026\021\255\227\177\000" [07:05:57.374] activitymsg = "waiting for 00000001", '0' , "6\000\177\000\000\224 \363\345\226U\000\000\360\340\027\346\226U\000\000\355\340\027\346\226U\000\000hb\025\276\375\177\000\000\353\340\027\346\226U\000" [07:05:57.374] path = "pg_wal/00000\000\000\000\000", '\060' , "\066\000pg_wal/RECOVERYHISTORY\000\000\b\000\000\000\000\000\000\066\000\000\000\000\000\000\000\000h%\363\345\000\000\000\062\000\000\000\000\000\000\000\000\b\000\000\000\000\000\000\000\334}\f\346\226U\000\000\b\000\000\000\000\000\000\000Y#\f\346\226U\000\000\210^\025\276\375\177\000\000U#\f\346\226U\000\000\224 \363\345\226U\000\000a#\f\346\226U\000\000Y#\f\346\226U\000\000\210^\025\276\375\177\000\000U#\f\346\226U\000\000\030\311\025\251\227\177\000\000\062*\363\345\226U", '\000' ... [07:05:57.374] fd = [07:05:57.374] __func__ = "XLogFileRead" [07:05:57.374] #7 0x00005596e5606881 in XLogFileReadAnyTLI (segno=6, emode=emode@entry=13, source=XLOG_FROM_ANY) at ../src/backend/access/transam/xlogrecovery.c:4334 [07:05:57.374] hent = [07:05:57.374] tli = 1 [07:05:57.374] cell__state = {l = , i = } [07:05:57.374] path = "\320c\025\276\375\177\000\000\200c\025\276\375\177\000\000\001\000\000\000\000\000\000\000\004\000\000\t", '\000' , "\302\035\250\346\226U\000\000@\036\250\346\226U\000\000Ta\025\276\375\177\000\000\030\340\027\346\226U\000\000\240\037\000\000\377\377\002\000@\036\250\346\226U\000\000ta\025\276\375\177\000\000'\340\027\346\226U\000\000@\036\250\346\226U\000\000\002\000\000\000\000\000\000\000\221:\363\345\226U\000\000va\025\276\375\177\000\000ta\025\276\375\177", '\000' , "\377\000\030\000\000\000\060\000\000\000Pa\025\276\375\177\000\000j;\363\345\226U\000\000\360N\025\276\000\000\000\000"... [07:05:57.374] cell = [07:05:57.374] fd = [07:05:57.374] tles = 0x5596e8656d38 [07:05:57.374] __func__ = "XLogFileReadAnyTLI" [07:05:57.374] #8 0x00005596e5607a7d in WaitForWALToBecomeAvailable (RecPtr=RecPtr@entry=100683560, randAccess=false, fetching_ckpt=false, tliRecPtr=tliRecPtr@entry=100683536, replayTLI=1, replayLSN=100683536, nonblocking=false) at ../src/backend/access/transam/xlogrecovery.c:3767 [07:05:57.374] oldSource = [07:05:57.374] startWalReceiver = false [07:05:57.374] last_fail_time = 763369366583780 [07:05:57.374] now = [07:05:57.374] streaming_reply_sent = true [07:05:57.374] __func__ = "WaitForWALToBecomeAvailable" [07:05:57.374] #9 0x00005596e56085b2 in XLogPageRead (xlogreader=0x5596e85f1ed0, targetPagePtr=100679680, reqLen=3880, targetRecPtr=100683536, readBuf=0x5596e8640d58 "\024\321\006") at ../src/backend/access/transam/xlogrecovery.c:3348 [07:05:57.374] private = 0x5596e85f1eb0 [07:05:57.374] emode = 15 [07:05:57.374] targetPageOff = 16384 [07:05:57.374] targetSegNo = 6 [07:05:57.374] r = [07:05:57.374] __func__ = "XLogPageRead" [07:05:57.374] #10 0x00005596e56109d7 in ReadPageInternal (state=state@entry=0x5596e85f1ed0, pageptr=pageptr@entry=100679680, reqLen=reqLen@entry=3880) at ../src/backend/access/transam/xlogreader.c:1060 [07:05:57.374] readLen = [07:05:57.374] targetPageOff = 16384 [07:05:57.374] targetSegNo = 6 [07:05:57.374] hdr = [07:05:57.374] #11 0x00005596e5612632 in XLogDecodeNextRecord (state=state@entry=0x5596e85f1ed0, nonblocking=nonblocking@entry=false) at ../src/backend/access/transam/xlogreader.c:597 [07:05:57.374] RecPtr = 100683536 [07:05:57.374] record = [07:05:57.374] targetPagePtr = 100679680 [07:05:57.374] randAccess = false [07:05:57.374] len = [07:05:57.374] total_len = [07:05:57.374] targetRecOff = 3856 [07:05:57.374] pageHeaderSize = [07:05:57.374] assembled = false [07:05:57.374] gotheader = [07:05:57.374] readOff = [07:05:57.374] decoded = 0x0 [07:05:57.374] errormsg = 0x5596e616c5a7 "LOG" [07:05:57.374] #12 0x00005596e56131d4 in XLogReadAhead (state=0x5596e85f1ed0, nonblocking=nonblocking@entry=false) at ../src/backend/access/transam/xlogreader.c:973 [07:05:57.374] result = [07:05:57.374] #13 0x00005596e55fdb4c in XLogPrefetcherNextBlock (pgsr_private=94106632006912, lsn=0x5596e86566d8) at ../src/backend/access/transam/xlogprefetcher.c:492 [07:05:57.374] nonblocking = [07:05:57.374] record = [07:05:57.374] prefetcher = 0x5596e85f2d00 [07:05:57.374] reader = 0x5596e85f1ed0 [07:05:57.374] replaying_lsn = 100683488 [07:05:57.374] __func__ = "XLogPrefetcherNextBlock" [07:05:57.374] #14 0x00005596e5601f15 in lrq_prefetch (lrq=) at ../src/backend/access/transam/xlogprefetcher.c:252 [07:05:57.374] No locals. [07:05:57.374] #15 lrq_complete_lsn (lsn=0, lrq=0x5596e8656608) at ../src/backend/access/transam/xlogprefetcher.c:290 [07:05:57.374] No locals. [07:05:57.374] #16 XLogPrefetcherReadRecord (prefetcher=prefetcher@entry=0x5596e85f2d00, errmsg=errmsg@entry=0x7ffdbe15a6e8) at ../src/backend/access/transam/xlogprefetcher.c:1037 [07:05:57.374] record = [07:05:57.374] replayed_up_to = 0 [07:05:57.374] #17 0x00005596e5608ddf in ReadRecord (xlogprefetcher=0x5596e85f2d00, emode=emode@entry=15, fetching_ckpt=fetching_ckpt@entry=false, replayTLI=1) at ../src/backend/access/transam/xlogrecovery.c:3148 [07:05:57.374] errormsg = 0x5596e85f26e0 "" [07:05:57.374] record = [07:05:57.394] xlogreader = 0x5596e85f1ed0 [07:05:57.394] private = [07:05:57.394] __func__ = "ReadRecord" [07:05:57.394] #18 0x00005596e560bf40 in PerformWalRecovery () at ../src/backend/access/transam/xlogrecovery.c:1829 [07:05:57.394] xtime = [07:05:57.394] ru0 = {tv = {tv_sec = 1710054161, tv_usec = 794527}, ru = {ru_utime = {tv_sec = 0, tv_usec = 1478}, ru_stime = {tv_sec = 0, tv_usec = 0}, {ru_maxrss = 16124, __ru_maxrss_word = 16124}, {ru_ixrss = 0, __ru_ixrss_word = 0}, {ru_idrss = 0, __ru_idrss_word = 0}, {ru_isrss = 0, __ru_isrss_word = 0}, {ru_minflt = 403, __ru_minflt_word = 403}, {ru_majflt = 5, __ru_majflt_word = 5}, {ru_nswap = 0, __ru_nswap_word = 0}, {ru_inblock = 0, __ru_inblock_word = 0}, {ru_oublock = 0, __ru_oublock_word = 0}, {ru_msgsnd = 0, __ru_msgsnd_word = 0}, {ru_msgrcv = 0, __ru_msgrcv_word = 0}, {ru_nsignals = 0, __ru_nsignals_word = 0}, {ru_nvcsw = 4, __ru_nvcsw_word = 4}, {ru_nivcsw = 3, __ru_nivcsw_word = 3}}} [07:05:57.394] record = 0x7f97a8a16f70 [07:05:57.394] reachedRecoveryTarget = false [07:05:57.394] replayTLI = 1 [07:05:57.394] __func__ = "PerformWalRecovery" [07:05:57.394] #19 0x00005596e55f5c23 in StartupXLOG () at ../src/backend/access/transam/xlog.c:5723 [07:05:57.394] Insert = [07:05:57.394] checkPoint = {redo = 83886120, ThisTimeLineID = 1, PrevTimeLineID = 1, fullPageWrites = true, nextXid = {value = 749}, nextOid = 24576, nextMulti = 1, nextMultiOffset = 0, oldestXid = 730, oldestXidDB = 1, oldestMulti = 1, oldestMultiDB = 1, time = 1710054161, oldestCommitTsXid = 0, newestCommitTsXid = 0, oldestActiveXid = 749} [07:05:57.394] wasShutdown = false [07:05:57.394] didCrash = [07:05:57.394] haveTblspcMap = false [07:05:57.394] haveBackupLabel = true [07:05:57.394] EndOfLog = [07:05:57.394] EndOfLogTLI = [07:05:57.394] newTLI = [07:05:57.394] performedWalRecovery = [07:05:57.394] endOfRecoveryInfo = [07:05:57.394] abortedRecPtr = [07:05:57.394] missingContrecPtr = [07:05:57.394] oldestActiveXID = [07:05:57.394] promoted = false [07:05:57.394] __func__ = "StartupXLOG" [07:05:57.394] #20 0x00005596e5aa5cda in StartupProcessMain () at ../src/backend/postmaster/startup.c:252 [07:05:57.394] No locals. [07:05:57.394] #21 0x00005596e5a9618f in AuxiliaryProcessMain (auxtype=auxtype@entry=B_STARTUP) at ../src/backend/postmaster/auxprocess.c:91 [07:05:57.394] __func__ = "AuxiliaryProcessMain" [07:05:57.394] #22 0x00005596e5aa0f6e in StartChildProcess (type=type@entry=B_STARTUP) at ../src/backend/postmaster/postmaster.c:5351 [07:05:57.394] pid = [07:05:57.394] __func__ = "StartChildProcess" [07:05:57.394] #23 0x00005596e5aa541a in PostmasterMain (argc=argc@entry=4, argv=argv@entry=0x5596e85f0d90) at ../src/backend/postmaster/postmaster.c:1460 [07:05:57.394] opt = [07:05:57.394] status = [07:05:57.394] userDoption = [07:05:57.394] listen_addr_saved = false [07:05:57.394] output_config_variable = [07:05:57.394] __func__ = "PostmasterMain" [07:05:57.394] #24 0x00005596e58d0791 in main (argc=4, argv=0x5596e85f0d90) at ../src/backend/main/main.c:197 [07:05:57.394] do_check_root = [07:05:57.394] [07:05:57.394] Thread 1 (LWP 39671): [07:05:57.394] #0 0x00007f97ab7c5fc7 in sigprocmask () from /lib/x86_64-linux-gnu/libc.so.6 [07:05:57.394] No symbol table info available. [07:05:57.394] #1 0x00007f97ab87713a in ?? () from /lib/x86_64-linux-gnu/libc.so.6 [07:05:57.394] No symbol table info available. [07:05:57.394] #2 0x00007f97ab888a6f in clone () from /lib/x86_64-linux-gnu/libc.so.6 [07:05:57.394] No symbol table info available.