[12:05:37.267] src/tools/ci/cores_backtrace.sh linux /tmp/cores [12:05:37.992] dumping /tmp/cores/postgres-3-21683.core for /tmp/cirrus-ci-build/build/tmp_install/usr/local/pgsql/bin/postgres [12:05:39.835] [New LWP 21683] [12:05:39.835] [New LWP 21563] [12:05:39.835] Downloading separate debug info for /lib/x86_64-linux-gnu/libm.so.6... [12:05:40.805] Downloading separate debug info for /lib/x86_64-linux-gnu/libdl.so.2... [12:05:41.735] Downloading separate debug info for /lib/x86_64-linux-gnu/librt.so.1... [12:05:42.957] Downloading separate debug info for /lib/x86_64-linux-gnu/libgssapi_krb5.so.2... [12:05:45.328] Downloading separate debug info for /lib/x86_64-linux-gnu/libicuuc.so.67... [12:05:45.328] Downloading separate debug info for /.cache/debuginfod_client/f871bbd529a02abd860f0d16b842b5b20234cb49/debuginfo... [12:05:47.213] Downloading separate debug info for /lib/x86_64-linux-gnu/libicui18n.so.67... [12:05:48.427] Downloading separate debug info for /lib/x86_64-linux-gnu/libldap_r-2.4.so.2... [12:05:50.083] Downloading separate debug info for /lib/x86_64-linux-gnu/libxml2.so.2... [12:05:51.185] Downloading separate debug info for /lib/x86_64-linux-gnu/liblz4.so.1... [12:05:51.981] Downloading separate debug info for /lib/x86_64-linux-gnu/libpam.so.0... [12:05:53.297] Downloading separate debug info for /lib/x86_64-linux-gnu/libssl.so.1.1... [12:05:54.990] Downloading separate debug info for /lib/x86_64-linux-gnu/libcrypto.so.1.1... [12:05:56.455] Downloading separate debug info for /lib/x86_64-linux-gnu/libsystemd.so.0... [12:05:57.368] Downloading separate debug info for /lib/x86_64-linux-gnu/libz.so.1... [12:05:58.909] Downloading separate debug info for /lib/x86_64-linux-gnu/libzstd.so.1... [12:06:00.479] Downloading separate debug info for /lib/x86_64-linux-gnu/libpthread.so.0... [12:06:00.952] [Thread debugging using libthread_db enabled] [12:06:00.952] Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1". [12:06:02.319] Downloading separate debug info for /lib64/ld-linux-x86-64.so.2... [12:06:03.713] Downloading separate debug info for /lib/x86_64-linux-gnu/libkrb5.so.3... [12:06:04.773] Downloading separate debug info for /lib/x86_64-linux-gnu/libk5crypto.so.3... [12:06:05.389] Downloading separate debug info for /lib/x86_64-linux-gnu/libcom_err.so.2... [12:06:06.154] Downloading separate debug info for /lib/x86_64-linux-gnu/libkrb5support.so.0... [12:06:08.444] Downloading separate debug info for /lib/x86_64-linux-gnu/libstdc++.so.6... [12:06:09.369] Downloading separate debug info for /lib/x86_64-linux-gnu/libgcc_s.so.1... [12:06:10.167] Downloading separate debug info for /lib/x86_64-linux-gnu/liblber-2.4.so.2... [12:06:11.493] Downloading separate debug info for /lib/x86_64-linux-gnu/libresolv.so.2... [12:06:12.629] Downloading separate debug info for /lib/x86_64-linux-gnu/libsasl2.so.2... [12:06:14.301] Downloading separate debug info for /lib/x86_64-linux-gnu/libgnutls.so.30... [12:06:15.367] Downloading separate debug info for /lib/x86_64-linux-gnu/liblzma.so.5... [12:06:16.164] Downloading separate debug info for /lib/x86_64-linux-gnu/libaudit.so.1... [12:06:17.548] Downloading separate debug info for /lib/x86_64-linux-gnu/libgcrypt.so.20... [12:06:18.165] Downloading separate debug info for /lib/x86_64-linux-gnu/libkeyutils.so.1... [12:06:19.703] Downloading separate debug info for /lib/x86_64-linux-gnu/libp11-kit.so.0... [12:06:20.466] Downloading separate debug info for /lib/x86_64-linux-gnu/libidn2.so.0... [12:06:21.690] Downloading separate debug info for /lib/x86_64-linux-gnu/libunistring.so.2... [12:06:22.478] Downloading separate debug info for /lib/x86_64-linux-gnu/libtasn1.so.6... [12:06:23.547] Downloading separate debug info for /lib/x86_64-linux-gnu/libnettle.so.8... [12:06:24.470] Downloading separate debug info for /lib/x86_64-linux-gnu/libhogweed.so.6... [12:06:25.687] Downloading separate debug info for /lib/x86_64-linux-gnu/libgmp.so.10... [12:06:26.329] Downloading separate debug info for /lib/x86_64-linux-gnu/libcap-ng.so.0... [12:06:27.250] Downloading separate debug info for /lib/x86_64-linux-gnu/libgpg-error.so.0... [12:06:28.013] Downloading separate debug info for /lib/x86_64-linux-gnu/libffi.so.7... [12:06:28.516] Core was generated by `postgres: cascade: startup recovering 000000020000000000000004 '. [12:06:28.516] Program terminated with signal SIGQUIT, Quit. [12:06:28.516] #0 0x00007f60be641fc7 in sigprocmask () from /lib/x86_64-linux-gnu/libc.so.6 [12:06:28.621] [Current thread is 1 (LWP 21683)] [12:06:28.758] [12:06:28.758] Thread 2 (Thread 0x7f60bbfd8a40 (LWP 21563)): [12:06:28.758] #0 0x00007f60be704a61 in clone () from /lib/x86_64-linux-gnu/libc.so.6 [12:06:28.758] No symbol table info available. [12:06:28.758] #1 0x00007f60be6f2e5e in ?? () from /lib/x86_64-linux-gnu/libc.so.6 [12:06:28.758] No symbol table info available. [12:06:28.758] #2 0x00007f60be6f3497 in ?? () from /lib/x86_64-linux-gnu/libc.so.6 [12:06:28.758] No symbol table info available. [12:06:28.758] #3 0x00007f60be6f2cdb in posix_spawn () from /lib/x86_64-linux-gnu/libc.so.6 [12:06:28.758] No symbol table info available. [12:06:28.758] #4 0x00007f60be64ea69 in ?? () from /lib/x86_64-linux-gnu/libc.so.6 [12:06:28.758] No symbol table info available. [12:06:28.758] #5 0x000055903135d024 in RestoreArchivedFile (path=path@entry=0x7ffe40f38f90 "", xlogfname=xlogfname@entry=0x7ffe40f38f50 "00000003.history", recovername=recovername@entry=0x559031e6eb90 "RECOVERYHISTORY", expectedSize=expectedSize@entry=0, cleanupEnabled=cleanupEnabled@entry=false) at ../src/backend/access/transam/xlogarchive.c:177 [12:06:28.758] xlogpath = "pg_wal/RECOVERYHISTORY\000\000\b\000\000\000\000\000\000\064\000\000\000\000\000\000\000\000D'\314\061\000\000\000\063Ѝ\363@\376\177\000\000\b\000\000\000\000\000\000\000l\263\346\061\220U\000\000\b\000\000\000\000\000\000\000\331W\346\061\220U\000\000x\216\363@\376\177\000\000\325W\346\061\220U\000\000p\"\314\061\220U\000\000\341W\346\061\220U\000\000\331W\346\061\220U\000\000x\216\363@\376\177\000\000\325W\346\061\220U\000\000\030\211\375\273`\177\000\000\016,\314\061\220U", '\000' , "\b", '\000' , "\060", '\000' , "\002\000\000\000\000\000\000\000"... [12:06:28.758] xlogRestoreCmd = 0x55903420ba08 "cp \"/tmp/cirrus-ci-build/build/testrun/recovery/025_stuck_on_old_timeline/data/t_025_stuck_on_old_timeline_primary_data/archives/00000003.history\" \"pg_wal/RECOVERYHISTORY\"" [12:06:28.758] lastRestartPointFname = '0' , "\000\353\346\061\220U\000\000\030\212\363@\376\177\000\000C\353\346\061\220U\000\000\030\211\375\273`\177\000\000\314,\314\061\220U", '\000' , "\b", '\000' , "\060", '\000' , "\002", '\000' ... [12:06:28.758] rc = [12:06:28.758] stat_buf = {st_dev = 3472113003596310384, st_ino = 206966894640, st_nlink = 34184295202646376, st_mode = 0, st_uid = 0, st_gid = 0, __pad0 = 0, st_rdev = 8, st_size = 5922071200885860208, st_blksize = 6366210006019162949, st_blocks = 94076968324940, st_atim = {tv_sec = 94077799113328, tv_nsec = 94077800830945}, st_mtim = {tv_sec = 3747088967773083609, tv_nsec = 140729988123080}, st_ctim = {tv_sec = 837179349, tv_nsec = 140053447543064}, __glibc_reserved = {8, 94077800854380, 140729988123160}} [12:06:28.758] restartSegNo = [12:06:28.758] restartRedoPtr = 94077799114564 [12:06:28.758] restartTli = 0 [12:06:28.758] __func__ = "RestoreArchivedFile" [12:06:28.758] #6 0x00005590313319e2 in existsTimeLineHistory (probeTLI=probeTLI@entry=3) at ../src/backend/access/transam/timeline.c:235 [12:06:28.758] path = '\000' , "\002\000\000\000\000\000\000\000#\277\066\061\220U\000\000@\223\363@\376\177\000\000\a\000\000\000\r\000\000\000\001\000\000\000\000\000\000\000\001\000\000\000\000\000\000\000\300\223\363@\376\177\000\000p\223\363@\376\177\000\000\001\000\000\000\000\000\000\000\004\000\000\t", '\000' , "D\221\363@\376\177\000\000XQ\362\061\220U\000\000\321\025\207\062\220U\000\000@\026\207\062\220U\000\000\240\037\000\000\377\377\002\000gQ\362\061\220U\000\000@\026\207\062\220U\000\000\002\000\000\000\000\000\000\000m<\314\061\220U\000\000f\221\363@\376\177\000\000d\221\363@\376\177", '\000' ... [12:06:28.758] histfname = "00000003.history", '\000' [12:06:28.758] fd = [12:06:28.758] __func__ = "existsTimeLineHistory" [12:06:28.758] #7 0x0000559031331acd in findNewestTimeLine (startTLI=startTLI@entry=2) at ../src/backend/access/transam/timeline.c:277 [12:06:28.758] newestTLI = 2 [12:06:28.758] probeTLI = 3 [12:06:28.758] #8 0x0000559031367fef in rescanLatestTimeLine (replayTLI=replayTLI@entry=2, replayLSN=replayLSN@entry=67211504) at ../src/backend/access/transam/xlogrecovery.c:4124 [12:06:28.758] newExpectedTLEs = [12:06:28.758] found = [12:06:28.758] cell = [12:06:28.758] newtarget = [12:06:28.758] oldtarget = 2 [12:06:28.758] currentTle = 0x0 [12:06:28.758] __func__ = "rescanLatestTimeLine" [12:06:28.758] #9 0x000055903136d19f in WaitForWALToBecomeAvailable (RecPtr=RecPtr@entry=67211528, randAccess=false, fetching_ckpt=false, tliRecPtr=tliRecPtr@entry=67211504, replayTLI=2, replayLSN=67211504, nonblocking=false) at ../src/backend/access/transam/xlogrecovery.c:3685 [12:06:28.758] oldSource = XLOG_FROM_STREAM [12:06:28.758] startWalReceiver = false [12:06:28.758] last_fail_time = 766065744808220 [12:06:28.758] now = [12:06:28.758] streaming_reply_sent = true [12:06:28.758] __func__ = "WaitForWALToBecomeAvailable" [12:06:28.758] #10 0x000055903136dbd5 in XLogPageRead (xlogreader=0x5590341b8fb0, targetPagePtr=67207168, reqLen=4360, targetRecPtr=67211504, readBuf=0x5590342081d8 "\025\321\006") at ../src/backend/access/transam/xlogrecovery.c:3362 [12:06:28.758] private = 0x5590341b8f90 [12:06:28.758] emode = 15 [12:06:28.758] targetPageOff = 98304 [12:06:28.758] targetSegNo = 4 [12:06:28.758] r = [12:06:28.758] __func__ = "XLogPageRead" [12:06:28.758] #11 0x0000559031376169 in ReadPageInternal (state=state@entry=0x5590341b8fb0, pageptr=pageptr@entry=67207168, reqLen=reqLen@entry=4360) at ../src/backend/access/transam/xlogreader.c:1060 [12:06:28.758] readLen = [12:06:28.758] targetPageOff = 98304 [12:06:28.758] targetSegNo = 4 [12:06:28.758] hdr = [12:06:28.758] #12 0x0000559031377dc4 in XLogDecodeNextRecord (state=state@entry=0x5590341b8fb0, nonblocking=nonblocking@entry=false) at ../src/backend/access/transam/xlogreader.c:597 [12:06:28.758] RecPtr = 67211504 [12:06:28.758] record = [12:06:28.758] targetPagePtr = 67207168 [12:06:28.758] randAccess = false [12:06:28.758] len = [12:06:28.758] total_len = [12:06:28.758] targetRecOff = 4336 [12:06:28.758] pageHeaderSize = [12:06:28.758] assembled = false [12:06:28.758] gotheader = [12:06:28.758] readOff = [12:06:28.758] decoded = 0x0 [12:06:28.758] errormsg = 0x559031f134e1 "LOG" [12:06:28.758] #13 0x0000559031378966 in XLogReadAhead (state=0x5590341b8fb0, nonblocking=nonblocking@entry=false) at ../src/backend/access/transam/xlogreader.c:973 [12:06:28.758] result = [12:06:28.758] #14 0x000055903136316f in XLogPrefetcherNextBlock (pgsr_private=94077837876704, lsn=0x55903421dd08) at ../src/backend/access/transam/xlogprefetcher.c:492 [12:06:28.758] nonblocking = [12:06:28.758] record = [12:06:28.758] prefetcher = 0x5590341b9de0 [12:06:28.758] reader = 0x5590341b8fb0 [12:06:28.758] replaying_lsn = 67211096 [12:06:28.758] __func__ = "XLogPrefetcherNextBlock" [12:06:28.758] #15 0x0000559031367538 in lrq_prefetch (lrq=) at ../src/backend/access/transam/xlogprefetcher.c:252 [12:06:28.758] No locals. [12:06:28.758] #16 lrq_complete_lsn (lsn=0, lrq=0x55903421da88) at ../src/backend/access/transam/xlogprefetcher.c:290 [12:06:28.758] No locals. [12:06:28.758] #17 XLogPrefetcherReadRecord (prefetcher=prefetcher@entry=0x5590341b9de0, errmsg=errmsg@entry=0x7ffe40f3d6d8) at ../src/backend/access/transam/xlogprefetcher.c:1037 [12:06:28.758] record = [12:06:28.758] replayed_up_to = 0 [12:06:28.758] #18 0x000055903136e402 in ReadRecord (xlogprefetcher=0x5590341b9de0, emode=emode@entry=15, fetching_ckpt=fetching_ckpt@entry=false, replayTLI=2) at ../src/backend/access/transam/xlogrecovery.c:3162 [12:06:28.758] errormsg = 0x5590341b97c0 "" [12:06:28.758] record = [12:06:28.758] xlogreader = 0x5590341b8fb0 [12:06:28.758] private = [12:06:28.758] __func__ = "ReadRecord" [12:06:28.758] #19 0x0000559031371626 in PerformWalRecovery () at ../src/backend/access/transam/xlogrecovery.c:1843 [12:06:28.758] xtime = [12:06:28.758] ru0 = {tv = {tv_sec = 1712750544, tv_usec = 274853}, ru = {ru_utime = {tv_sec = 0, tv_usec = 1996}, ru_stime = {tv_sec = 0, tv_usec = 0}, {ru_maxrss = 16668, __ru_maxrss_word = 16668}, {ru_ixrss = 0, __ru_ixrss_word = 0}, {ru_idrss = 0, __ru_idrss_word = 0}, {ru_isrss = 0, __ru_isrss_word = 0}, {ru_minflt = 407, __ru_minflt_word = 407}, {ru_majflt = 5, __ru_majflt_word = 5}, {ru_nswap = 0, __ru_nswap_word = 0}, {ru_inblock = 0, __ru_inblock_word = 0}, {ru_oublock = 0, __ru_oublock_word = 0}, {ru_msgsnd = 0, __ru_msgsnd_word = 0}, {ru_msgrcv = 0, __ru_msgrcv_word = 0}, {ru_nsignals = 0, __ru_nsignals_word = 0}, {ru_nvcsw = 11, __ru_nvcsw_word = 11}, {ru_nivcsw = 1, __ru_nivcsw_word = 1}}} [12:06:28.779] record = 0x7f60bb8abe50 [12:06:28.779] reachedRecoveryTarget = false [12:06:28.779] replayTLI = 2 [12:06:28.779] __func__ = "PerformWalRecovery" [12:06:28.779] #20 0x000055903135afb2 in StartupXLOG () at ../src/backend/access/transam/xlog.c:5807 [12:06:28.779] Insert = [12:06:28.779] checkPoint = {redo = 33554472, ThisTimeLineID = 1, PrevTimeLineID = 1, fullPageWrites = true, nextXid = {value = 739}, nextOid = 14202, nextMulti = 1, nextMultiOffset = 0, oldestXid = 731, oldestXidDB = 1, oldestMulti = 1, oldestMultiDB = 1, time = 1712750542, oldestCommitTsXid = 0, newestCommitTsXid = 0, oldestActiveXid = 739} [12:06:28.779] wasShutdown = false [12:06:28.779] didCrash = [12:06:28.779] haveTblspcMap = false [12:06:28.779] haveBackupLabel = true [12:06:28.779] EndOfLog = [12:06:28.779] EndOfLogTLI = [12:06:28.779] newTLI = [12:06:28.779] performedWalRecovery = [12:06:28.779] endOfRecoveryInfo = [12:06:28.779] abortedRecPtr = [12:06:28.779] missingContrecPtr = [12:06:28.779] oldestActiveXID = [12:06:28.779] promoted = false [12:06:28.779] __func__ = "StartupXLOG" [12:06:28.779] #21 0x0000559031820f5b in StartupProcessMain (startup_data=, startup_data_len=) at ../src/backend/postmaster/startup.c:258 [12:06:28.779] No locals. [12:06:28.779] #22 0x0000559031818572 in postmaster_child_launch (child_type=child_type@entry=B_STARTUP, startup_data=startup_data@entry=0x0, startup_data_len=startup_data_len@entry=0, client_sock=client_sock@entry=0x0) at ../src/backend/postmaster/launch_backend.c:265 [12:06:28.779] pid = [12:06:28.779] #23 0x000055903181a5f0 in StartChildProcess (type=type@entry=B_STARTUP) at ../src/backend/postmaster/postmaster.c:3928 [12:06:28.779] pid = [12:06:28.779] __func__ = "StartChildProcess" [12:06:28.779] #24 0x00005590318206ba in PostmasterMain (argc=argc@entry=4, argv=argv@entry=0x5590341b7e70) at ../src/backend/postmaster/postmaster.c:1364 [12:06:28.779] opt = [12:06:28.779] status = [12:06:28.779] userDoption = [12:06:28.779] listen_addr_saved = false [12:06:28.779] output_config_variable = [12:06:28.779] __func__ = "PostmasterMain" [12:06:28.779] #25 0x00005590316413ef in main (argc=4, argv=0x5590341b7e70) at ../src/backend/main/main.c:197 [12:06:28.779] do_check_root = [12:06:28.779] [12:06:28.779] Thread 1 (LWP 21683): [12:06:28.779] #0 0x00007f60be641fc7 in sigprocmask () from /lib/x86_64-linux-gnu/libc.so.6 [12:06:28.779] No symbol table info available. [12:06:28.779] #1 0x00007f60be6f313a in ?? () from /lib/x86_64-linux-gnu/libc.so.6 [12:06:28.779] No symbol table info available. [12:06:28.779] #2 0x00007f60be704a6f in clone () from /lib/x86_64-linux-gnu/libc.so.6 [12:06:28.779] No symbol table info available.