Unitary Boundary Pairs for Isometric Operators in Pontryagin Spaces and Generalized Coresolvents

Abstract

An isometric operator V in a Pontryagin space ${{{\mathfrak {H}}}}$ is called standard, if its domain and the range are nondegenerate subspaces in ${{{\mathfrak {H}}}}$. A description of coresolvents for standard isometric operators is known and basic underlying concepts that appear in the literature are unitary colligations and characteristic functions. In the present paper generalized coresolvents of non-standard Pontryagin space isometric operators are described. The methods used in this paper rely on a new general notion of boundary pairs introduced for isometric operators in a Pontryagin space setting. Even in the Hilbert space case this notion generalizes the earlier concept of boundary triples for isometric operators and offers an alternative approach to study operator valued Schur functions without any additional invertibility requirements appearing in the ordinary boundary triple approach.

1 Introduction

Extension theory for standard symmetric and isometric operators in Pontryagin spaces was first developed by Iokhvidov and Kreĭn in [30], generalized resolvents of such operators were described by Kreĭn and Langer in [37,38,39]. Following [28] we will use the notion standard for an isometric operator V in a Pontryagin space ${{{\mathfrak {H}}}}$, if its domain $\mathrm{dom\,}V$ and the range $\mathrm{ran\,}V$ are nondegenerate subspaces in ${{{\mathfrak {H}}}}$. In this case every unitary extension of V can be obtained in pretty much the same way as in the case of Hilbert space isometric operator. Similarly, the extension theory and the theory of generalized coresolvents of standard isometric operators in Kreĭn spaces was built by Dijksma et al. in [28]. For a nonstandard isometric operator in a Pontryagin space, description of its regular (resp. nonregular) generalized coresolvents in Pontryagin spaces without growth (resp. with growth) of negative index was given by Sorjonen [46] (resp. by Nitz [43, 44]). However, the proof in [44] is not so convincing, as it becomes quite complicated and contains some gaps.

Another approach to the extension theory of symmetric operators in Hilbert spaces is based on the concept of abstract boundary value introduced by Calkin [13] and later formalized in the notion of boundary value space in [33, 35] (or ordinary boundary triple in [26]). In [25] with each boundary triple there was associated an analytic object—abstract Weyl function which allows to carry out spectral analysis of extensions of symmetric operators. In the case of a Hilbert space isometric operator (and more generally for a dual pair of operators) the notions of a boundary triple and a corresponding Weyl function were introduced in [41, 42]. These notions, when generalized to the indefinite case in [5], proved to be an adequate language in the extension theory of nonstandard isometric operator V in a Pontryagin space, since they allowed to give full description of generalized coresolvents of V. However, the method proposed in [5] is restricted to the case of regular generalized coresolvents, which have minimal realizations in Pontryagin spaces ${{\widetilde{{\mathfrak {H}}}} }$ with the same negative index as ${{\mathfrak {H}}}$, and does not work for generalized coresolvents of V which have minimal realizations in Pontryagin spaces ${{\widetilde{{\mathfrak {H}}}} }$ with bigger negative indices.

This difficulty can be prevented by using an appropriate notion of boundary pairs, which extend the concept of ordinary boundary triples. In the case of symmetric operators in Hilbert spaces an extension of ordinary boundary triples, a so-called generalized boundary triple, was introduced and studied in [26]. This notion was further generalized in [19] to the notion of a unitary boundary pair (called therein as a boundary relation), which can be applied to study generalized resolvents of symmetric operators [20, 21] and various general classes of boundary value problems for ordinary and partial differential operators, see [21,22,23,24]. In particular, in [19] it was shown that every Nevanlinna pair (or Nevanlinna family of holomorphic relations) can be realized as the Weyl family of some unitary boundary pair, and in [18, 20] this notion was used to get a new proof of Kreĭn formula for generalized resolvents of symmetric operators via the coupling method developed therein. In [8] the notion of unitary boundary pair was introduced for symmetric operators in Pontryagin spaces and it was shown that every generalized Nevanlinna pair, allowing a finite negative index for the associated Nevanlinna kernel, can be realized as the Weyl family of such a unitary boundary pair.

In this paper a new notion of a unitary boundary pair with an associated Weyl function is introduced and studied in the setting of isometric operators in Pontryagin spaces. In particular, it is shown in Sect. 3 how a certain subclass of unitary boundary pairs is connected to unitary colligations (see [1]) and, moreover, that the Weyl functions associated with that subclass of unitary boundary pairs actually coincide with characteristic functions of the corresponding unitary colligations; see Theorems 3.10, 3.11. Furthermore, using some transformation results, being motivated by [1], it is also shown that every operator valued generalized Schur function (not necessarily holomorphic at the origin) can still be realized as the Weyl function of some unitary boundary pair for an isometric operator V in a Pontryagin space; see Theorems 3.15, 3.17. These two theorems show that the present notion of a unitary boundary pair for isometric operators in a Pontryagin (as well as in the classical Hilbert) space setting is a natural object to realize and study generalized (or standard) Schur functions as their Weyl functions. In particular, these new notions complement and extend the approach, which relies on characteristic functions of unitary colligations being associated with the special case stated in Theorem 3.11.

After these characteristic results on unitary boundary pairs and their Weyl function for isometric operators we study in Sect. 4 some spectral properties of proper extensions of V and find a formula for their canonical coresolvents, see Theorem 4.2, and then with these preparations prove an analog of Kreĭn formula for the generalized coresolvents of the isometric operator V. This latter problem is solved via the coupling method, where we consider a coupled unitary boundary pair as a direct sum of an ordinary boundary triple and a unitary boundary pair and then derive the formula for generalized coresolvents from the formula for canonical coresolvents associated with the coupled boundary pair.

2 Preliminaries

2.1 Indefinite Inner Product Spaces

A linear space ${{\mathfrak {H}}}$ endowed with an inner product $[\cdot ,\cdot ]_{{{\mathfrak {H}}}}$ is called an inner product space, see [4, 11]. A vector $f\in {{\mathfrak {H}}}$ is called positive (resp. negative or neutral), if $[f,f]_{{{\mathfrak {H}}}}>0$ (resp. $[f,f]_{{{\mathfrak {H}}}}<0$ or $[f,f]_{{{\mathfrak {H}}}}=0$). A subspace ${{{\mathfrak {L}}}}\subset {{\mathfrak {H}}}$ is called positive (resp. negative or neutral), if every vector $f\in {{\mathfrak {L}}}\setminus \{0\}$ is positive (resp. negative or neutral). The orthogonal complement of a subspace ${{{\mathfrak {L}}}}$ is denoted by ${{{\mathfrak {L}}}}^{[\perp ]}$.

An inner product space ${{\mathfrak {H}}}$ is called a Kreĭn space, if it admits a fundamental decomposition

$$\begin{aligned} {{\mathfrak {H}}}={{\mathfrak {H}}}_+[+]{{\mathfrak {H}}}_- \end{aligned}$$

(2.1)

as an orthogonal sum of a positive subspace ${{\mathfrak {H}}}_+$ and a negative subspace ${{\mathfrak {H}}}_-$. The operator $J=P_+-P_-$, where $P_{\pm }$ are orthogonal projections in ${{\mathfrak {H}}}$ onto ${{\mathfrak {H}}}_\pm$, is called the fundamental symmetry of ${{\mathfrak {H}}}$. We will use the notation $({{\mathfrak {H}}},J)$ for the Kreĭn space ${{\mathfrak {H}}}$ with the fundamental symmetry J. A Kreĭn space $({{\mathfrak {H}}},J)$ with a finite negative index $\kappa _-({{\mathfrak {H}}}):=\mathrm{dim\,}{{\mathfrak {H}}}_-$ is called a Pontryagin space.

Every closed subspace ${{{\mathfrak {D}}}}$ of a Pontryagin space $({{\mathfrak {H}}},J)$ admits the following decomposition

$$\begin{aligned} {{{\mathfrak {D}}}}={{{\mathfrak {D}}}}_0\dot{+} {{{\mathfrak {D}}}}_+ \dot{+} {{{\mathfrak {D}}}}_-, \end{aligned}$$

(2.2)

where ${{{\mathfrak {D}}}}_0={{{\mathfrak {D}}}}\cap {{{\mathfrak {D}}}}^{[\perp ]}$ is a neutral subspace (the isotropic part of ${{{\mathfrak {D}}}}$) and ${{{\mathfrak {D}}}}_+$ and ${{{\mathfrak {D}}}}_-$ are closed (uniformly) positive and negative subspaces of $({{\mathfrak {H}}},J)$; see e.g. [11, Theorem IX.2.5]. We will need the following slightly modified version of this statement.

Lemma 2.1

Every linear subspace ${{\mathfrak {T}}}$ of a Pontryagin space $({{\mathfrak {H}}},J)$ admits the following decomposition

$$\begin{aligned} {{\mathfrak {T}}}={{\mathfrak {T}}}_+ \dot{+} {{\mathfrak {T}}}_1, \end{aligned}$$

(2.3)

where ${{\mathfrak {T}}}_+$ is a positive subspace of $({{\mathfrak {H}}},J)$, such that $\overline{{{\mathfrak {T}}}_+}$ is a maximal positive subspace of $\overline{{{\mathfrak {T}}}}$, and ${{{\mathfrak {T}}}_1}$ is a k-dimensional subspace of $({{\mathfrak {H}}},J)$, where $k=\mathrm{dim\,}\overline{{{\mathfrak {T}}}}/ \overline{{{\mathfrak {T}}}_+}$.

Proof

Let ${{{\mathfrak {D}}}}$ be the closure of ${{\mathfrak {T}}}$ in ${{\mathfrak {H}}}$ and decompose ${{\mathfrak {D}}}$ as in (2.2),

$$\begin{aligned} {{{\mathfrak {D}}}}={{{\mathfrak {D}}}}_0\dot{+} {{{\mathfrak {D}}}}_- \dot{+} {{{\mathfrak {D}}}}_+, \end{aligned}$$

where ${{{\mathfrak {D}}}}_0={{{\mathfrak {D}}}}\cap {{{\mathfrak {D}}}}^{[\perp ]}$, ${{{\mathfrak {D}}}}_-$, and ${{{\mathfrak {D}}}}_+$ are closed neutral, negative, and positive subspaces of $({{\mathfrak {H}}},J)$, respectively. Since ${{\mathfrak {T}}}$ is a dense subset of ${{{\mathfrak {D}}}}$ and the subspaces ${{{\mathfrak {D}}}}_0$ and ${{{\mathfrak {D}}}}_-$ are finite dimensional, ${{\mathfrak {T}}}$ has a dense intersection with ${{{\mathfrak {D}}}}_+$,

$$\begin{aligned} \overline{{{\mathfrak {T}}}\cap {{{\mathfrak {D}}}}_+}={{{\mathfrak {D}}}}_+; \end{aligned}$$

(2.4)

see e.g. [32, Lemma 2.1]. Denote ${{{\mathfrak {T}}}}_+:={{\mathfrak {T}}}\cap {{{\mathfrak {D}}}}_+$ and let $k=\mathrm{dim\,}({{{\mathfrak {D}}}}_0\dot{+} {{{\mathfrak {D}}}}_-)$. Since $\overline{{{\mathfrak {T}}}}={{\mathfrak {D}}}$ one concludes that there exists a k-dimensional subspace ${{{\mathfrak {T}}}}_1\subset {{\mathfrak {T}}}\setminus {{\mathfrak {D}}}_+$. The closed subspace ${{{\mathfrak {T}}}}_1\subset {{\mathfrak {T}}}$ decomposes ${{\mathfrak {T}}}$ and (2.4) together with a dimension argument leads to

$$\begin{aligned} \overline{{{{\mathfrak {T}}}}_+ \dot{+} {{{\mathfrak {T}}}}_1}= \overline{{{{\mathfrak {T}}}}_+} \dot{+}{{{\mathfrak {T}}}}_1= {{\mathfrak {D}}}_+ \dot{+} {{{\mathfrak {T}}}}_1={{\mathfrak {D}}}=\overline{{{\mathfrak {T}}}}. \end{aligned}$$

The equality $\overline{{{\mathfrak {T}}}}={{{\mathfrak {T}}}}_1 \dot{+} \overline{{{{\mathfrak {T}}}}_+}$ combined with (2.4) yields the decomposition (2.3) for ${{\mathfrak {T}}}$.

$\square$

2.2 Linear Relations in Kreĭn Spaces

2.2 Linear Relations in Kreĭn Spaces

Let $({{\mathfrak {H}}}_1,J_{{{\mathfrak {H}}}_1})$ and $({{\mathfrak {H}}}_2,J_{{{\mathfrak {H}}}_2})$ be two Kreĭn spaces. A linear relation T from ${{\mathfrak {H}}}_1$ to ${{\mathfrak {H}}}_2$ is a linear subspace of ${{\mathfrak {H}}}_1 \times {{\mathfrak {H}}}_2$, see e.g. [2]. Often a linear operator $T:{{\mathfrak {H}}}_1\rightarrow {{\mathfrak {H}}}_2$ will be identified with its graph

$$\begin{aligned} gr T:=\{\,\{f,Tf\}:\,f\in \mathrm{dom\,}T\,\}. \end{aligned}$$

For a linear relation T from ${{\mathfrak {H}}}_1$ to ${{\mathfrak {H}}}_2$ the symbols $\mathrm{dom\,}T$, ${\mathrm{ker\,}\,}T$, $\mathrm{ran\,}T$, and $\mathrm{mul\,}T$ stand for the domain, kernel, range, and multivalued part, respectively. The inverse $T^{-1}$ is a relation from ${{\mathfrak {H}}}_2$ to ${{\mathfrak {H}}}_1$ defined by $\{\,\{f',f\}:\,\{f,f'\}\in T\,\}$. Denote by $\rho (T)$ the resolvent set of T, by $\sigma (T)$ the spectrum of T and by $\sigma _p(T)$ (resp. $\sigma _c(T)$, $\sigma _r(T)$) the point (resp. continuous, residual) spectrum of T. The adjoint $T^{[*]}$ is the closed linear relation from ${{\mathfrak {H}}}_2$ to ${{\mathfrak {H}}}_1$ defined by (see [9])

$$\begin{aligned} T^{[*]}=\{\,\{h,k\} \in {{\mathfrak {H}}}_2 \times {{\mathfrak {H}}}_1 :\, [k,f]_{{{\mathfrak {H}}}_1}=[h,g]_{{{\mathfrak {H}}}_2}, \, \{f,g\}\in T \,\}. \end{aligned}$$

(2.5)

The following equalities are obvious from (2.5)

$$\begin{aligned} (\mathrm{dom\,}T)^{[\perp ]}=\mathrm{mul\,}T^{[*]}, \quad (\mathrm{ran\,}T)^{[\perp ]}={\mathrm{ker\,}\,}T^{[*]}. \end{aligned}$$

(2.6)

A linear relation T from ${{\mathfrak {H}}}_1$ to ${{\mathfrak {H}}}_2$ is called isometric (resp. contractive or expanding), if for every $\{f,g\}\in T$ one has

$$\begin{aligned}{}[g,g]_{{{\mathfrak {H}}}_2}=[f,f]_{{{\mathfrak {H}}}_1} \quad \left( resp. \, [g,g]_{{{\mathfrak {H}}}_2}\le [f,f]_{{{\mathfrak {H}}}_1} or \, [g,g]_{{{\mathfrak {H}}}_2}\ge [f,f]_{{{\mathfrak {H}}}_1}\right) .\nonumber \\ \end{aligned}$$

(2.7)

It follows from (2.5) and (2.7) that T is isometric, if $T^{-1}\subseteq T^{[*]}$. A linear relation T from ${{\mathfrak {H}}}_1$ to ${{\mathfrak {H}}}_2$ is called unitary, if $T^{-1}= T^{[*]}$, [45]. Moreover, T is said to be a standard unitary operator if $\mathrm{dom\,}T={{\mathfrak {H}}}_1$ and $\mathrm{ran\,}T={{\mathfrak {H}}}_2$. For an isometric linear relation T one obtains from $T^{-1}\subseteq T^{[*]}$ and the identities (2.6) that

$$\begin{aligned} {\mathrm{ker\,}\,}T \subseteq (\mathrm{dom\,}T)^{[\perp ]}, \quad \mathrm{mul\,}T \subseteq (\mathrm{ran\,}T)^{[\perp ]}. \end{aligned}$$

(2.8)

For a unitary linear relation the following statements hold, see [45, Theorem 2].

Proposition 2.2

Let T be a unitary relation from the Kreĭn space $({{\mathfrak {H}}}_1,J_{{{\mathfrak {H}}}_1})$ to the Kreĭn space $({{\mathfrak {H}}}_2,J_{{{\mathfrak {H}}}_2})$. Then:

• (i) $\mathrm{dom\,}T$ is closed if and only if $\mathrm{ran\,}T$ is closed;

• (ii) the following equalities hold:

  $$\begin{aligned} {\mathrm{ker\,}\,}T=(\mathrm{dom\,}T)^{[\perp ]},\quad \mathrm{mul\,}T=(\mathrm{ran\,}T)^{[\perp ]}. \end{aligned}$$

  (2.9)

Denote

$$\begin{aligned} {{\mathbb {D}}}=\{\lambda \in {{\mathbb {C}}}:\, |\lambda |<1 \}, \quad {{\mathbb {D}}}_e=\{\lambda \in {{\mathbb {C}}}:\, |\lambda |>1 \}, \quad {{\mathbb {T}}}=\{\lambda \in {{\mathbb {C}}}:\, |\lambda |=1 \}.\nonumber \\ \end{aligned}$$

(2.10)

If V is a single-valued closed isometric operator in a Pontryagin space ${{\mathfrak {H}}}$ then the subspaces $\mathrm{ran\,}(V-\lambda I)$ are closed for every $\lambda \in {{\mathbb {D}}}\cup {{\mathbb {D}}}_e$, see e.g. [40, Section 1.3] or Lemma 2.4 below, and each of the sets $\sigma _p(V)\cap {{\mathbb {D}}}$ and $\sigma _p(V)\cap {{\mathbb {D}}}_e$ consist of at most $\kappa =\kappa _-({{\mathfrak {H}}})$ eigenvalues, see e.g. [31, p. 49 Corollary 2]. Denote by ${{\mathfrak {N}}}_\lambda$ the defect subspace of V:

$$\begin{aligned} {{\mathfrak {N}}}_\lambda :={{\mathfrak {H}}}[-]\mathrm{ran\,}(I-\overline{\lambda } V), \quad \lambda \in {{\mathbb {D}}}\cup {{\mathbb {D}}}_e. \end{aligned}$$

Then

$$\begin{aligned} {{\mathfrak {N}}}_\lambda ={\mathrm{ker\,}\,}(V^{-[*]}-\lambda I)=\{f_\lambda :(f_\lambda ,\lambda f_\lambda )^T\in V^{-{[*]}}\}. \end{aligned}$$

As is known, see [31, Theorem 6.1], the numbers $\mathrm{dim\,}{{\mathfrak {N}}}_\lambda$ take a constant value $n_+(V)$ for all $\lambda \in {{\mathbb {D}}}\setminus {\overline{\sigma _p(V)}}^{-1}$, and $n_-(V)$ for all $\lambda \in {{\mathbb {D}}}_e\setminus {\overline{\sigma _p(V)}}^{-1}$. The numbers $n_\pm (V)$ are called the defect numbers of V.

Definition 2.3

The isometric operator V in ${{{\mathfrak {H}}}}$ is called simple, if $\sigma _p(V)\setminus {{\mathbb {T}}}=\emptyset$ and

$$\begin{aligned} \overline{\mathrm{span\,}}\{{{\mathfrak {N}}}_\lambda :\lambda \in {{\mathbb {D}}}\cup {{\mathbb {D}}}_e\}={{{\mathfrak {H}}}}. \end{aligned}$$

In the case of Pontryagin spaces some further results on isometric and unitary relations can be established. For any isometric relation T between two Kreĭn spaces it is clear that ${\mathrm{ker\,}\,}T$ and $\mathrm{mul\,}T$ are neutral subspaces. Therefore, in a Pontryagin space ${\mathrm{ker\,}\,}T$ and $\mathrm{mul\,}T$ are necessarily finite dimensional. If T is closed then ${\mathrm{ker\,}\,}T$ and $\mathrm{mul\,}T$ are also closed. In Pontryagin spaces the following stronger result is true.

Lemma 2.4

Let T be a closed isometric relation from the Pontryagin space $({{\mathfrak {H}}}_1,J_{{{\mathfrak {H}}}_1})$ to the Pontryagin space $({{\mathfrak {H}}}_2,J_{{{\mathfrak {H}}}_2})$. Then the domain and the range of T are closed.

Proof

The isometry of T means that $T^{-1}\subseteq T^{[*]}$. Taking inverses one gets $T=(T^{-1})^{-1}\subseteq T^{-[*]}$, i.e., T and $T^{-1}$ are simultaneously isometric. Therefore, to prove the statement it suffices to prove that the range of T is a closed subspace in ${{\mathfrak {H}}}_2$, since T is closed precisely when its inverse $T^{-1}$ is closed and clearly $\mathrm{dom\,}T=\mathrm{ran\,}T^{-1}$.

Now let $\mathrm{dom\,}T$ be decomposed as in Lemma 2.1,

$$\begin{aligned} \mathrm{dom\,}T= {{{\mathfrak {T}}}}_+ \dot{+} {{{\mathfrak {T}}}}_1, \end{aligned}$$

(2.11)

so that ${{\mathfrak {D}}}_+:=\overline{{{{\mathfrak {T}}}}_+}$ is a maximal uniformly positive subspace of ${{\mathfrak {D}}}:=\mathrm{\overline{dom}\,}T$. Next introduce the restriction of (the graph of) T by setting

$$\begin{aligned} T_+:=T \cap \left( {{\mathfrak {D}}}_+ \times {{\mathfrak {H}}}_2\right) . \end{aligned}$$

Then $T_+$ is closed and as a restriction of T it is also an isometric relation from $({{\mathfrak {H}}}_1,J_{{{\mathfrak {H}}}_1})$ to $({{\mathfrak {H}}}_2,J_{{{\mathfrak {H}}}_2})$. Moreover, $\mathrm{dom\,}T_+={{{\mathfrak {T}}}}_+\subseteq {{\mathfrak {D}}}_+$ is a uniformly positive subspace. This implies that for all $\{f,f'\}\in T_+$ and some $\delta >0$,

$$\begin{aligned}{}[f',f']_2=[f,f]_1\ge \delta \Vert f\Vert _1^2, \end{aligned}$$

which shows that ${\mathrm{ker\,}\,}T_+=\{0\}$, so that $(T_+)^{-1}$ is an isometric operator, and, moreover,

$$\begin{aligned} \Vert (T_+)^{-1}f'\Vert _1^2=\Vert f\Vert _1^2\le \delta ^{-1} [f,f]_1 \le \delta ^{-1} \Vert f\Vert _1^2. \end{aligned}$$

Therefore, the closed isometric operator $(T_+)^{-1}$ is also bounded. Consequently, $\mathrm{ran\,}T_+=\mathrm{dom\,}(T_+)^{-1}$ is a closed subspace in ${{\mathfrak {H}}}_2$. On the other hand, since $\mathrm{dom\,}T$ admits the decomposition (2.11), where ${{{\mathfrak {T}}}}_1$ is finite dimensional and $\mathrm{mul\,}T=\mathrm{mul\,}T_+$ (also finite dimensional), one concludes that

$$\begin{aligned} \mathrm{ran\,}T=T({{{\mathfrak {T}}}}_1) + \mathrm{ran\,}T_+ \end{aligned}$$

as a finite dimensional extension of the closed subspace $\mathrm{ran\,}T_+$ is a closed subspace of ${{\mathfrak {H}}}_2$. This completes the proof. $\square$

Lemma 2.4 can be seen as an extension of [11, Theorem IX.3.1]. It is known e.g. from [11, Theorem IX.3.2] and [31, Theorems 6.2, 6.3]) that if T is an isometric operator in a Pontryagin space such that $\mathrm{\overline{ran}\,}T$ (resp. $\mathrm{\overline{dom}\,}T$) is a nondegenerate subspace, then T (resp. $T^{-1}$) is continuous. The next lemma contains main properties of isometric relations acting between two Pontryagin spaces.

Lemma 2.5

For an isometric relation T from the Pontryagin space $({{\mathfrak {H}}}_1,J_{{{\mathfrak {H}}}_1})$ to the Pontryagin space $({{\mathfrak {H}}}_2,J_{{{\mathfrak {H}}}_2})$ the following statements hold:

• (i) If $\mathrm{\overline{ran}\,}T$ (resp. $\mathrm{\overline{dom}\,}T$) is a nondegenerate subspace of $({{\mathfrak {H}}}_2,J_{{{\mathfrak {H}}}_2})$, then T (resp. $T^{-1}$) is a continuous operator.

• (ii) If T is densely defined then $\kappa _-({{\mathfrak {H}}}_1)\le \kappa _-({{\mathfrak {H}}}_2)$ and if, in addition, $\kappa _-({{\mathfrak {H}}}_1)=\kappa _-({{\mathfrak {H}}}_2)$, then $\mathrm{ran\,}\overline{T}$ is a closed nondegenerate subspace of ${{\mathfrak {H}}}_2$ and, moreover, T and $T^{-1}$ are continuous operators.

• (iii) If $\mathrm{ran\,}T$ is dense in ${{\mathfrak {H}}}_2$ then $\kappa _-({{\mathfrak {H}}}_1)\ge \kappa _-({{\mathfrak {H}}}_2)$ and if, in addition, $\kappa _-({{\mathfrak {H}}}_1)=\kappa _-({{\mathfrak {H}}}_2)$, then $\mathrm{dom\,}\overline{T}$ is a closed nondegenerate subspace of ${{\mathfrak {H}}}_1$ and, moreover, T and $T^{-1}$ are continuous operators.

• (iv) If the relation T is unitary and ${\mathrm{ker\,}\,}T=\mathrm{mul\,}T=\{0\}$, then $\kappa _-({{\mathfrak {H}}}_1)=\kappa _-({{\mathfrak {H}}}_2)$ and T is a standard unitary operator.

• (v) If the relation T is unitary and $\kappa _-({{\mathfrak {H}}}_1)=\kappa _-({{\mathfrak {H}}}_2)$, then

  $$\begin{aligned} \mathrm{mul\,}T=\{0\}\quad \Longleftrightarrow \quad {\mathrm{ker\,}\,}T=\{0\}. \end{aligned}$$

  (2.12)

  In particular, if $\kappa _-({{\mathfrak {H}}}_1)=\kappa _-({{\mathfrak {H}}}_2)$ then a unitary relation T is an operator if and only if it is a standard unitary operator.

Proof

• (i) By assumption $T\subseteq T^{-[*]}$ and $\overline{T}\subseteq T^{-[*]}$, i.e., $\overline{T}$ is also isometric. Hence, if $\mathrm{\overline{ran}\,}T$ is nondegenerate, (2.8) implies

  $$\begin{aligned} \mathrm{mul\,}\overline{T}\subset \mathrm{\overline{ran}\,}T\cap (\mathrm{\overline{ran}\,}T)^{[\perp ]}=\{0\}. \end{aligned}$$

  Thus T is a closable operator, which by Lemma 2.4 and the closed graph theorem implies that $\overline{T}$ and, therefore, also T is continuous. Similarly it is seen that

  $$\begin{aligned} {\mathrm{ker\,}\,}\overline{T}\subseteq \mathrm{\overline{dom}\,}T\cap (\mathrm{\overline{dom}\,}T)^{[\perp ]}=\{0\}, \end{aligned}$$

  if $\mathrm{\overline{dom}\,}T$ is nondegenerate, and then $\overline{T}^{-1}$ and $T^{-1}$ are continuous.

• (ii) Since $\mathrm{\overline{dom}\,}T={{\mathfrak {H}}}_1$ is nondegenerate, $T^{-1}$ is a continuous operator by item (i). On the other hand $\mathrm{dom\,}T$, as a dense subspace of ${{\mathfrak {H}}}_1$, contains a negative subspace ${{\mathcal {D}}}_-\subset \mathrm{dom\,}T$ of dimension $\kappa _-({{\mathfrak {H}}}_1)$; see [11, Theorem IX.1.4]. Then also $\mathrm{ran\,}T$ contains a negative subspace of the same dimension and hence $\kappa _-({{\mathfrak {H}}}_2)\ge \kappa _-({{\mathfrak {H}}}_1)$. Now assume that $\kappa _-({{\mathfrak {H}}}_1)=\kappa _-({{\mathfrak {H}}}_2)$. Then $\mathrm{\overline{ran}\,}T=\mathrm{\overline{ran}\,}\overline{T}$ is necessarily a nondegenerate subspace of the Pontryagin space $({{\mathfrak {H}}}_2,J_{{{\mathfrak {H}}}_2})$, see e.g. [11, Lemma II.10.5], and (i) shows that T is a continuous operator.

• (iii) This follows by applying (ii) to $T^{-1}$, which is also an isometric relation.

• (iv) If T is unitary then the conditions ${\mathrm{ker\,}\,}T=\mathrm{mul\,}T=\{0\}$ are equivalent to $\mathrm{\overline{dom}\,}T={{\mathfrak {H}}}_1$ and $\mathrm{\overline{ran}\,}T={{\mathfrak {H}}}_2$; see (2.9) in Proposition 2.2. Now the assertions follow from (ii) and (iii).

• (v) Let $\kappa _-({{\mathfrak {H}}}_1)=\kappa _-({{\mathfrak {H}}}_2)$ and let T be unitary. By symmetry it suffices to prove one implication in (2.12), say, “$\Rightarrow$”. The condition $\mathrm{mul\,}T=\{0\}$ is equivalent to $\mathrm{\overline{ran}\,}T={{\mathfrak {H}}}_2$; see (2.9). Now item (iii) shows that T and $T^{-1}$ are continuous operators. Thus, in particular, ${\mathrm{ker\,}\,}T=\{0\}$ and (2.9) together with Lemma 2.4 shows that $\mathrm{dom\,}T={{\mathfrak {H}}}_1$ and $\mathrm{ran\,}T={{\mathfrak {H}}}_2$. $\square$

2.3 Operator Colligations

Let ${{\mathfrak {H}}}$ be a Pontryagin space, ${{\mathfrak {L}}}_1$ and ${{\mathfrak {L}}}_2$ be Hilbert spaces. The set of bounded everywhere defined operators from ${{\mathfrak {L}}}_1$ to ${{\mathfrak {L}}}_2$ is denoted by ${\mathbf {B}}({{\mathfrak {L}}}_1,{{\mathfrak {L}}}_2)$, ${\mathbf {B}}({{\mathfrak {L}}}_1):={\mathbf {B}}({{\mathfrak {L}}}_1,{{\mathfrak {L}}}_1)$. Let U be a bounded operator from ${{{\mathfrak {H}}}}\oplus {{{\mathfrak {L}}}}_1$ to ${{{\mathfrak {H}}}}\oplus {{{\mathfrak {L}}}}_2$ represented in the block form

$$\begin{aligned} U=\left( \begin{array}{cc} T &{} F \\ G &{} H \\ \end{array} \right) :\begin{pmatrix} {{{\mathfrak {H}}}} \\ {{{\mathfrak {L}}}}_1 \end{pmatrix}\rightarrow \begin{pmatrix} {{{\mathfrak {H}}}} \\ {{{\mathfrak {L}}}}_2 \end{pmatrix}. \end{aligned}$$

The quadruple $({{{\mathfrak {H}}}},{{{\mathfrak {L}}}}_1,{{{\mathfrak {L}}}}_2, U)$ is called a colligation, ${{\mathfrak {H}}}$ is the state space, ${{{\mathfrak {L}}}}_1$ and ${{{\mathfrak {L}}}}_2$ are the incoming and the outgoing spaces, T is the main operator and U is called the connecting operator of the colligation. The colligation $({{{\mathfrak {H}}}},{{{\mathfrak {L}}}}_1,{{{\mathfrak {L}}}}_2, U)$ is called unitary, if U is a unitary operator from ${{{\mathfrak {H}}}}\oplus {{{\mathfrak {L}}}}_1$ to ${{{\mathfrak {H}}}}\oplus {{{\mathfrak {L}}}}_2$. The colligation $({{{\mathfrak {H}}}},{{{\mathfrak {L}}}}_2,{{{\mathfrak {L}}}}_1, U^{[*]})$ is called adjoint to the colligation $({{{\mathfrak {H}}}},{{{\mathfrak {L}}}}_1,{{{\mathfrak {L}}}}_2, U)$; cf. [1, 12].

Components of a unitary colligation satisfy the following identities

$$\begin{aligned} \begin{aligned}&T^{[*]}T+G^{[*]}G=I_{{{{\mathfrak {H}}}}},\ F^{[*]}F+H^*H=I_{{{\mathfrak {L}}}_1},\ T^{[*]}F+G^{[*]}H=0,\\&TT^{[*]}+FF^{[*]}=I_{{{{\mathfrak {H}}}}},\ GG^{[*]}+HH^*=I_{{{\mathfrak {L}}}_2},\ TG^{[*]}+FH^*=0, \end{aligned} \end{aligned}$$

(2.13)

which are equivalent to the identities

$$\begin{aligned} U^{[*]}U=I_{{{{\mathfrak {H}}}}\oplus {{{\mathfrak {L}}}}_1},\quad UU^{[*]}=I_{{{{\mathfrak {H}}}}\oplus {{{\mathfrak {L}}}}_2}. \end{aligned}$$

A colligation $({{{\mathfrak {H}}}},{{{\mathfrak {L}}}}_1,{{{\mathfrak {L}}}}_2,U)$ is said to be closely connected, if

$$\begin{aligned} {{\mathfrak {H}}}=\overline{span }\,\{\mathrm{ran\,}(T^mF),\mathrm{ran\,}((T^{[*]})^nG^{[*]}):\,m,n\ge 0\}. \end{aligned}$$

(2.14)

A unitary colligation is closely connected if and only if the operator U has no nontrivial reducing subspaces. The operator valued function

$$\begin{aligned} \Theta _\Delta (\lambda ):=H+\lambda G(I-\lambda T)^{-1}F,\quad 1/\lambda \in \rho (T), \end{aligned}$$

is called the characteristic function of the unitary colligation $({{{\mathfrak {H}}}},{{{\mathfrak {L}}}}_1,{{{\mathfrak {L}}}}_2,U)$.

Recall, see e.g. [1], that a ${\mathbf {B}}({{\mathfrak {L}}}_1,{{\mathfrak {L}}}_2)$-valued function $\Theta (\lambda )$ is said to belong to the generalized Schur class ${{\mathcal {S}}}_{\kappa }^0({{\mathfrak {L}}}_1,{{\mathfrak {L}}}_2)$ if it is holomorphic in a neighborhood $\Omega$ of 0 and the kernel

$$\begin{aligned} {{\mathsf {K}}}_\omega ^\Theta (\lambda )= \frac{I-\Theta (\lambda )\Theta (\omega )^*}{1-\lambda \overline{\omega }} \end{aligned}$$

(2.15)

has $\kappa$ negative squares in $\Omega \times \Omega$, i.e. for any finite set of points $\omega _1,\dots ,\omega _n$ in $\Omega$ and vectors $f_1,\dots ,f_n$ in ${{\mathfrak {L}}}_2$, the Hermitian matrix

$$\begin{aligned} \left( ({{\mathsf {K}}}_{\omega _i}^\Theta (\omega _j)f_j,f_i)_{{{\mathfrak {L}}}_2}\right) _{i,j=1}^n \end{aligned}$$

(2.16)

has at most $\kappa$ negative eigenvalues, and for some choice of $\omega _1,\dots ,\omega _n$ in $\Omega$ and $f_1,\dots ,f_n$ in ${{\mathfrak {L}}}_2$ the matrix (2.16) has exactly $\kappa$ negative eigenvalues.

As is known, see [1], the characteristic function of a closely connected unitary colligation belongs to the generalized Schur class ${{\mathcal {S}}}_{\kappa }^0({{\mathfrak {L}}}_1,{{\mathfrak {L}}}_2)$, where $\kappa =\kappa _-({{\mathfrak {H}}})$. Moreover, the converse is also true; see e.g. [1, Theorem 2.3.1].

Theorem 2.6

Let ${{\mathfrak {L}}}_1$ and ${{\mathfrak {L}}}_2$ be Hilbert spaces and let S $(\lambda )$ belong to the generalized Schur class ${{\mathcal {S}}}_{\kappa }^0({{\mathfrak {L}}}_1,{{\mathfrak {L}}}_2)$. Then there exists a closely connected unitary colligation $({{{\mathfrak {H}}}},{{{\mathfrak {L}}}}_1,{{{\mathfrak {L}}}}_2,U)$, such that the corresponding characteristic function $\Theta _\Delta$ $(\lambda )$ coincides with S $(\lambda )$ in a neighborhood $\Omega$ of 0.

In what follows a ${\mathbf {B}}({{\mathfrak {L}}}_1,{{\mathfrak {L}}}_2)$-valued function $\Theta (\cdot )$ holomorphic in some open subset $\Omega \subset {{\mathbb {D}}}$ is said to belong to the generalized Schur class ${{\mathcal {S}}}_{\kappa }({{\mathfrak {L}}}_1,{{\mathfrak {L}}}_2)$, if the kernel (2.15) has $\kappa$ negative squares in $\Omega \times \Omega$. In particular, we do not require that $0\in \Omega$, which implies that characteristic functions of unitary colligations used in Theorem 2.6 are not sufficient to give a realization for all functions $\Theta (\cdot )$ from the class ${{\mathcal {S}}}_{\kappa }({{\mathfrak {L}}}_1,{{\mathfrak {L}}}_2)$ . In the next section we introduce the notions of a unitary boundary pair for an isometric operator and an associated Weyl function as a replacement for unitary colligations and their characteristic functions. These new notions allow to realize an arbitrary operator function from the class ${{\mathcal {S}}}_{\kappa }({{\mathfrak {L}}}_1,{{\mathfrak {L}}}_2)$ as the Weyl function of a Pontryagin space isometric operator, corresponding to some unitary boundary pair.

3 Unitary Boundary Pairs for Isometric Operators

3.1 Unitary Boundary Pairs and the Main Transform

Let ${{{\mathfrak {H}}}}$ be a Pontryagin space with the negative index $\kappa$ and the fundamental symmetry $J_{{\mathfrak {H}}}$ and let ${{{\mathfrak {L}}}}_1$ and ${{{\mathfrak {L}}}}_2$ be Hilbert spaces. In this section we introduce the notion of a unitary boundary pair for an isometric operator V: here, and in what follows, V is assumed to be closed.

For this purpose equip the Hilbert spaces ${{{\mathfrak {H}}}}^2$ and ${{{\mathfrak {L}}}}={{{\mathfrak {L}}}}_1\times {{{\mathfrak {L}}}}_2$ with the indefinite inner products by the formulas

$$\begin{aligned}{}[{{\widehat{f}} },{{\widehat{g}} }]_{{{{\mathfrak {H}}}}^2}=( J_{{{{\mathfrak {H}}}^2}}{{\widehat{f}} },{{\widehat{g}} })_{{{\mathfrak {H}}}^2},\quad [{{\widehat{u}} },{{\widehat{v}} }]_{{{{\mathfrak {L}}}}}=\left( J_{{{{\mathfrak {L}}}}}{{\widehat{u}} },{{\widehat{v}} }\right) _{{{\mathfrak {L}}}}, \end{aligned}$$

where

$$\begin{aligned} J_{{{{\mathfrak {H}}}}^2}=\left( \begin{array}{cc} J_{{{{\mathfrak {H}}}}} &{} 0 \\ 0 &{} -J_{{{{\mathfrak {H}}}}} \\ \end{array} \right) ,\qquad J_{{{{\mathfrak {L}}}}}=\left( \begin{array}{cc} I_{{{{\mathfrak {L}}}}_1} &{} 0 \\ 0 &{} -I_{{{{\mathfrak {L}}}}_2} \\ \end{array} \right) , \end{aligned}$$

and ${{\widehat{f}} }=\{f,f'\}, {{\widehat{g}} }=\{g,g'\}\in {{\mathfrak {H}}}^2$, ${{\widehat{u}} }=\{u_1,u_2\}, {{\widehat{v}} }=\{v_1,v_2\}\in {{\mathfrak {L}}}$. Then $({{{\mathfrak {H}}}}^2, J_{{{{\mathfrak {H}}}}^2})$ and $({{{\mathfrak {L}}}}, J_{{{{\mathfrak {L}}}}})$ are Kreĭn spaces. In particular, for ${{\mathfrak {V}}}\subset {{{\mathfrak {H}}}}^2$ the linear set ${{\mathfrak {V}}}^{[\perp ]}$ in the Kreĭn space $({{{\mathfrak {H}}}}^2, J_{{{{\mathfrak {H}}}}^2})$ can be characterized as follows

$$\begin{aligned} {{\widehat{g}} } \in {{\mathfrak {V}}}^{[\perp ]} \quad \Longleftrightarrow \quad [{{\widehat{f}} },{{\widehat{g}} }]_{{{{\mathfrak {H}}}}^2}=0 \text { for all } {{\widehat{f}} }\in {{\mathfrak {V}}}\quad \Longleftrightarrow \quad {{\widehat{g}} } \in {{\mathfrak {V}}}^{-[*]}. \end{aligned}$$

(3.1)

Definition 3.1

Let ${{{\mathfrak {L}}}}_1$ and ${{{\mathfrak {L}}}}_2$ be Hilbert spaces, let V be a closed isometric operator (or isometric relation) in ${{{\mathfrak {H}}}}$ and let $\Gamma$ be a linear relation ${{{\mathfrak {H}}}}^2\rightarrow {{{\mathfrak {L}}}}$, where ${{{\mathfrak {L}}}}:={{{\mathfrak {L}}}}_1\times {{{\mathfrak {L}}}}_2$.

The pair $({{{\mathfrak {L}}}},\Gamma )$ will be called a unitary boundary pair for V if:

• (1) $V={\mathrm{ker\,}\,}\Gamma$ and for all $\{{{\widehat{f}} }, {{\widehat{u}} }\},\{{{\widehat{g}} }, {{\widehat{v}} }\}\in \Gamma$ the following identity holds

  $$\begin{aligned}{}[f,g]_{{{{\mathfrak {H}}}}}-[f',g']_{{{{\mathfrak {H}}}}}=(u_1,v_1)_{{{{\mathfrak {L}}}}_1}-(u_2,v_2)_{{{{\mathfrak {L}}}}_2}; \end{aligned}$$

  (3.2)

• (2) $\Gamma$ is maximal in the sense that if $\{{{\widehat{g}} }, {{\widehat{v}} }\}\in {{{\mathfrak {H}}}}^2\times {{{\mathfrak {L}}}}$ satisfies (3.2) for all $\{{{\widehat{f}} },{{\widehat{u}} }\}\in \Gamma$, then $\{{{\widehat{g}} }, {{\widehat{v}} }\}\in \Gamma$. Here

  $$\begin{aligned} \{{{\widehat{f}} }, {{\widehat{u}} }\}= \left\{ \begin{pmatrix} f \\ f' \end{pmatrix}, \begin{pmatrix} u_1 \\ u_2 \end{pmatrix} \right\} ,\quad \{{{\widehat{g}} }, {{\widehat{v}} }\}=\left\{ \begin{pmatrix} g \\ g' \end{pmatrix}, \begin{pmatrix} v_1 \\ v_2 \end{pmatrix} \right\} \in {{{\mathfrak {H}}}}^2\times {{{\mathfrak {L}}}}. \end{aligned}$$

Item (1) of Definition 3.1 means that $\Gamma$ is an isometric linear relation from the Kreĭn space $({{{\mathfrak {H}}}}^2, J_{{{{\mathfrak {H}}}}^2})$ to the Kreĭn space $({{{\mathfrak {L}}}},J_{{{{\mathfrak {L}}}}})$, while items (1) and (2) together mean that $\Gamma$ is unitary.

Application of Proposition 2.2 to a unitary boundary pair leads to the following statement.

Proposition 3.2

Let $({{{\mathfrak {L}}}},\Gamma )$ be a unitary boundary pair for V. Then:

• (i) $V_*:=\mathrm{dom\,}\Gamma$ is dense in $V^{-[*]}$;

• (ii) $\mathrm{ran\,}\Gamma$ is dense in ${{{\mathfrak {L}}}}$ if and only if $\mathrm{mul\,}\Gamma =\{0\}$;

• (iii) $\mathrm{ran\,}\Gamma ={{{\mathfrak {L}}}}$ if and only if $\mathrm{dom\,}\Gamma =V^{-[*]}$ and $\mathrm{mul\,}\Gamma =\{0\}$.

Define the components $\Gamma _1$ and $\Gamma _2$ of $\Gamma$ by

$$\begin{aligned} \Gamma _1:= & {} \left\{ \{{\widehat{f}}, u_1 \}: \left\{ {\widehat{f}}, \begin{pmatrix} u_1 \\ u_2 \end{pmatrix} \right\} \in \Gamma \quad \text {for some} \quad u_2\in {{{\mathfrak {L}}}}_2 \right\} ; \end{aligned}$$

(3.3)

$$\begin{aligned} \Gamma _2:= & {} \left\{ \{{\widehat{f}},u_2 \}: \left\{ {\widehat{f}}, \begin{pmatrix} u_1 \\ u_2 \end{pmatrix} \right\} \in \Gamma \quad \text {for some} \quad u_1 \in {{{\mathfrak {L}}}}_1 \right\} . \end{aligned}$$

(3.4)

In the case that $\Gamma _1$ and $\Gamma _2$ are single-valued, i.e. $\mathrm{mul\,}\Gamma _1=\mathrm{mul\,}\Gamma _2=0$ and $\mathrm{dom\,}\Gamma =V^{-{[*]}}$ and $\mathrm{ran\,}\Gamma ={{{\mathfrak {L}}}}_1\times {{{\mathfrak {L}}}}_2$ the collection $\{{{{\mathfrak {L}}}}_1\times {{{\mathfrak {L}}}}_2,\Gamma _1,\Gamma _2\}$ is called an ordinary boundary triple for the isometric operator V. For a Hilbert space isometric operator the corresponding notion was introduced and studied in [41, 42] as a boundary triple for the dual pair $(V,V^{-1})$.

For ordinary boundary triples an application of the closed graph theorem shows that the component map**s $\Gamma _1$ and $\Gamma _2$ are bounded. However, for a general unitary boundary pair $({{{\mathfrak {L}}}},\Gamma )$ the map**s $\Gamma _1$ and $\Gamma _2$ need not be bounded or single-valued. With $\Gamma _1$ and $\Gamma _2$ one associates the extensions $V_1$ and $V_2$ of V by the equalities

$$\begin{aligned} V_1:={\mathrm{ker\,}\,}\Gamma _1,\quad V_2:={\mathrm{ker\,}\,}\Gamma _2. \end{aligned}$$

(3.5)

It follows from the identity (3.2) that $V_1$ is an expanding linear relation and $V_2$ is a contractive linear relation in the Pontryagin space ${{\mathfrak {H}}}$. Moreover, it is clear from (3.2) that the sets ${\mathrm{ker\,}\,}V_1\setminus {\mathrm{ker\,}\,}V$ and $\mathrm{mul\,}V_2\,(=\mathrm{mul\,}V_2\setminus \mathrm{mul\,}V)$ consist of negative vectors in ${{\mathfrak {H}}}$ while the sets ${\mathrm{ker\,}\,}V_2\setminus {\mathrm{ker\,}\,}V$ and $\mathrm{mul\,}V_1\,(=\mathrm{mul\,}V_1\setminus \mathrm{mul\,}V)$ consist of positive vectors of ${{\mathfrak {H}}}$.

Let ${{\mathfrak {H}}}_j:={{{\mathfrak {H}}}}\times {{{\mathfrak {L}}}}_j$ $(j=1,2)$ be a Pontryagin space with the inner product

$$\begin{aligned} \left[ \begin{pmatrix} f \\ u \end{pmatrix},\begin{pmatrix} f \\ u \end{pmatrix}\right] _{{{\mathfrak {H}}}_j}=[f,f]_{{\mathfrak {H}}}+\Vert u\Vert _{{{\mathfrak {L}}}_j}^2, \quad f\in {{\mathfrak {H}}},\,u\in {{\mathfrak {L}}}_j,\,j=1,2. \end{aligned}$$

In establishing some properties of unitary boundary pairs it is useful to connect the unitary relation $\Gamma$ which acts between two Kreĭn spaces to another unitary relation that acts between two Pontryagin spaces, since unitary relations between Pontryagin spaces have simpler structure. For this purpose we introduce the following transform from the Kreĭn space $\left( {{{\mathfrak {H}}}}^2\oplus {{{\mathfrak {L}}}},J_{{{{\mathfrak {H}}}}^2}\oplus (-J_{{\mathfrak {L}}})\right)$ to the Kreĭn space $\left( {{{\mathfrak {H}}}_2}\oplus {{{\mathfrak {H}}}_1},(J_{{{{\mathfrak {H}}}_2}})\oplus (-J_{{{{\mathfrak {H}}}_1}})\right)$ by

$$\begin{aligned} {{\mathcal {J}}}:\left\{ \begin{pmatrix} f \\ f' \end{pmatrix}, \begin{pmatrix} u_1 \\ u_2 \end{pmatrix} \right\} \mapsto \left\{ \begin{pmatrix} f \\ u_2 \end{pmatrix}, \begin{pmatrix} f' \\ u_1 \end{pmatrix} \right\} , \quad f,f'\in {{{\mathfrak {H}}}} \text { and } u_1\in {{{\mathfrak {L}}}}_1,\, u_2\in {{{\mathfrak {L}}}}_2 \end{aligned}$$

It establishes a one-to-one correspondence between (closed) linear relations $\Gamma$ from $({{{\mathfrak {H}}}}^2, J_{{{{\mathfrak {H}}}_2}})$ to $({{\mathfrak {L}}}, J_{{\mathfrak {L}}})$ and (closed) linear relations ${{\mathcal {U}}}$ from ${{{\mathfrak {H}}}_2}$ to ${{{\mathfrak {H}}}_1}$ via

$$\begin{aligned} \Gamma \mapsto {{\mathcal {U}}}:={{\mathcal {J}}}(\Gamma )= \left\{ \, \left\{ \begin{pmatrix} f \\ u_2 \end{pmatrix}, \begin{pmatrix} f' \\ u_1 \end{pmatrix} \right\} :\, \left\{ \begin{pmatrix} f \\ f' \end{pmatrix}, \begin{pmatrix} u_1 \\ u_2 \end{pmatrix} \right\} \in \Gamma \,\right\} . \end{aligned}$$

(3.6)

The linear relation ${{\mathcal {U}}}$ will be called the main transform of $\Gamma$; cf. [19] for the case of symmetric operators. In the following lemma, which is an analog of [19, Proposition 2.10], some basic properties of the transform ${{\mathcal {J}}}$ are given.

Lemma 3.3

Let the linear relation $\Gamma$ from $({{{\mathfrak {H}}}}^2, J_{{{{\mathfrak {H}}}}^2})$ to $({{{\mathfrak {L}}}}, J_{{{{\mathfrak {L}}}}})$ and the linear relation ${{\mathcal {U}}}$ from ${{{\mathfrak {H}}}}\times {{{\mathfrak {L}}}}_2$ to ${{{\mathfrak {H}}}}\times {{{\mathfrak {L}}}}_1$ be connected by ${{\mathcal {U}}}={{\mathcal {J}}}(\Gamma )$. Then

$$\begin{aligned} {{\mathcal {U}}}^{-[*]}={{\mathcal {J}}}(\Gamma ^{-[*]}). \end{aligned}$$

(3.7)

Moreover, the transform ${{\mathcal {J}}}$ establishes a one-to-one correspondence between isometric (unitary, contractive, expanding) relations $\Gamma$ from $({{{\mathfrak {H}}}}^2, J_{{{{\mathfrak {H}}}}^2})$ to $({{{\mathfrak {L}}}}, J_{{{{\mathfrak {L}}}}})$ and isometric (unitary, contractive, expanding) relations ${{\mathcal {U}}}$ from ${{{\mathfrak {H}}}}\times {{{\mathfrak {L}}}}_2$ to ${{{\mathfrak {H}}}}\times {{{\mathfrak {L}}}}_1$.

Proof

It is straightforward to check that for all elements of the form

$$\begin{aligned} \left\{ \begin{pmatrix}f \\ f' \end{pmatrix},\begin{pmatrix}u_1 \\ u_2 \end{pmatrix}\right\} , \left\{ \begin{pmatrix}g \\ g' \end{pmatrix},\begin{pmatrix}v_1 \\ v_2 \end{pmatrix}\right\} \in \begin{pmatrix}{{{\mathfrak {H}}}} \\ {{{\mathfrak {H}}}} \end{pmatrix}\times \begin{pmatrix}{{{\mathfrak {L}}}}_1 \\ {{{\mathfrak {L}}}}_2 \end{pmatrix}, \end{aligned}$$

the following identity is satisfied:

$$\begin{aligned} \left[ \begin{pmatrix}f \\ u_2 \end{pmatrix},\begin{pmatrix}g \\ v_2 \end{pmatrix}\right] _{{{\mathfrak {H}}}_2} -\left[ \begin{pmatrix}f' \\ u_1 \end{pmatrix},\begin{pmatrix}g' \\ v_1 \end{pmatrix}\right] _{{{\mathfrak {H}}}_1} = \left[ \begin{pmatrix}f \\ f' \end{pmatrix},\begin{pmatrix}g \\ g' \end{pmatrix}\right] _{{{\mathfrak {H}}}^2} -\left[ \begin{pmatrix}u_1 \\ u_2 \end{pmatrix},\begin{pmatrix}v_1 \\ v_2 \end{pmatrix}\right] _{{\mathfrak {L}}}.\nonumber \\ \end{aligned}$$

(3.8)

In view of (3.1) this identity implies the equivalence

$$\begin{aligned} \left\{ \begin{pmatrix}f \\ u_2 \end{pmatrix},\begin{pmatrix}f' \\ u_1 \end{pmatrix}\right\} \in {{\mathcal {U}}}^{-[*]}\Longleftrightarrow \left\{ \begin{pmatrix}f \\ f' \end{pmatrix},\begin{pmatrix}u_1 \\ u_2 \end{pmatrix}\right\} \in \Gamma ^{-[*]} \end{aligned}$$

which leads to identity (3.7). It follows from (3.7) that

$$\begin{aligned} {{\mathcal {U}}}^{-1}\subseteq {{\mathcal {U}}}^{[*]}\Longleftrightarrow \Gamma ^{-1}\subseteq \Gamma ^{[*]}, \quad {{\mathcal {U}}}^{-1}={{\mathcal {U}}}^{[*]}\Longleftrightarrow \Gamma ^{-1}=\Gamma ^{[*]}, \end{aligned}$$

i.e., ${{\mathcal {U}}}$ is isometric (unitary) precisely when $\Gamma$ is isometric (resp. unitary). The connection between contractive (expanding) relations $\Gamma$ and ${{\mathcal {U}}}$ is clear from (3.8). $\square$

The next proposition contains the basic properties of $\Gamma _1$, $\Gamma _2$ and $V_1$, $V_2$ for a unitary boundary pair $({{{\mathfrak {L}}}},\Gamma )$.

Proposition 3.4

Let $({{{\mathfrak {L}}}},\Gamma )$ be a unitary boundary pair for V, let $\Gamma _1$, $\Gamma _2$ and $V_1$, $V_2$ be defined by (3.3), (3.4) and (3.5), and let ${{\mathcal {U}}}$ be the main transform of $\Gamma$. Then:

• (i) $\Gamma _1$ and $\Gamma _2$ are closed;

• (ii) $\mathrm{ran\,}\Gamma _1={{\mathfrak {L}}}_1$ and $\mathrm{ran\,}\Gamma _2={{\mathfrak {L}}}_2$;

• (iii) $\mathrm{mul\,}\Gamma _j=P_{{{\mathfrak {L}}}_j}(\mathrm{mul\,}\Gamma )$, $j=1,2$, and the following equivalences hold:

  $$\begin{aligned} \mathrm{mul\,}\Gamma _1=\{0\}\quad \Longleftrightarrow \quad \mathrm{mul\,}\Gamma _2=\{0\} \quad \Longleftrightarrow \quad \mathrm{mul\,}\Gamma =\{0\}; \end{aligned}$$

• (iv) the extensions $V_1$ and $V_2$ of V are closed, $V_1\subseteq V_2^{-[*]}$ and $V_2\subseteq V_1^{-[*]}$;

• (v) the following equivalences hold:

  $$\begin{aligned} \mathrm{mul\,}V_2= & {} \{0\}\Longleftrightarrow \mathrm{mul\,}{{\mathcal {U}}}=\{0\};\\ {\mathrm{ker\,}\,}V_1= & {} \{0\}\Longleftrightarrow {\mathrm{ker\,}\,}{{\mathcal {U}}}=\{0\};\\ \mathrm{mul\,}V_2= & {} \{0\} \Longleftrightarrow {\mathrm{ker\,}\,}V_1=\{0\}. \end{aligned}$$

  If one of the sets appearing in the above equivalences is trivial, then the main transform ${{\mathcal {U}}}$ from ${{\mathfrak {H}}}\times {{\mathfrak {L}}}_2$ to ${{\mathfrak {H}}}\times {{\mathfrak {L}}}_1$ of $\Gamma$ is a standard unitary operator.

Proof

• (i) To prove that $\Gamma _1$ is closed assume that $\{{{\widehat{f}} }_n,{{\widehat{u}} }_n\}\in \Gamma$ such that ${{\widehat{f}} }_n\rightarrow {{\widehat{f}} }\in {{\mathfrak {H}}}^2$ and $u_{1,n}\rightarrow u_1\in {{\mathfrak {L}}}_1$. Then (3.2) gives

  $$\begin{aligned}&{[}f_n-f_m,f_n-f_m]_{{{{\mathfrak {H}}}}}-[f_n'-f_m',f_n'-f_m']_{{{{\mathfrak {H}}}}}\\&\quad =\Vert u_{1,n}-u_{1,m}\Vert ^2_{{{{\mathfrak {L}}}}_1}-\Vert u_{2,n}-u_{2,m}\Vert ^2_{{{{\mathfrak {L}}}}_2} \end{aligned}$$

  and letting $n,m\rightarrow \infty$ one concludes that $\Vert u_{2,n}-u_{2,m}\Vert _{{{{\mathfrak {L}}}}_2}\rightarrow 0$. As a Cauchy sequence $(u_{2,n})$ converges to some element $u_2$ in ${{\mathfrak {L}}}_2$. This means that $\{{{\widehat{f}} }_n,{{\widehat{u}} }_n\}\rightarrow \{{{\widehat{f}} },{{\widehat{u}} }\}$ and, since $\Gamma$ is closed as a unitary relation, $\{{{\widehat{f}} },{{\widehat{u}} }\}\in \Gamma$ and thus $\{{{\widehat{f}} },u_1\}\in \Gamma _1$. This proves that $\Gamma _1$ is closed. Similarly one proves that $\Gamma _2$ is closed.

• (ii) First it is shown that $\mathrm{ran\,}\Gamma _1$ is a closed subspace of ${{\mathfrak {L}}}_1$. For this consider the main transform ${{\mathcal {U}}}={{\mathcal {J}}}(\Gamma )$. By Lemma 3.3${{\mathcal {U}}}$ is a unitary relation between the Pontryagin spaces ${{\mathfrak {H}}}\times {{\mathfrak {L}}}_2$ and ${{\mathfrak {H}}}\times {{\mathfrak {L}}}_1$. Moreover, by Lemma 2.4$\mathrm{ran\,}{{\mathcal {U}}}$ is closed and Proposition 2.2 shows that $\mathrm{mul\,}{{\mathcal {U}}}$ is the isotropic part of $\mathrm{ran\,}{{\mathcal {U}}}$. Therefore $\mathrm{mul\,}{{\mathcal {U}}}$ is a closed finite dimensional subspace of ${{\mathfrak {H}}}\times {{\mathfrak {L}}}_1$ and thus also the co-dimension k of $\mathrm{ran\,}{{\mathcal {U}}}$ is finite ($k\le \kappa _-({{\mathfrak {H}}}))$. Let ${{\mathfrak {M}}}$ be any k-dimensional subspace such that $\mathrm{ran\,}{{\mathcal {U}}}\dot{+}{{\mathfrak {M}}}={{\mathfrak {H}}}\times {{\mathfrak {L}}}_1$ and let $P_1$ be the orthogonal projection from ${{\mathfrak {H}}}\times {{\mathfrak {L}}}_1$ onto ${{\mathfrak {L}}}_1$. Then

  $$\begin{aligned} {{\mathfrak {L}}}_1=P_1(\mathrm{ran\,}{{\mathcal {U}}}\dot{+}{{\mathfrak {M}}})=P_1\mathrm{ran\,}{{\mathcal {U}}}+ P_1{{\mathfrak {M}}}, \end{aligned}$$

  and here $\mathrm{dim\,}P_1{{\mathfrak {M}}}\le k$, which implies that $P_1\mathrm{ran\,}{{\mathcal {U}}}=\mathrm{ran\,}\Gamma _1$ is closed.

  To see that $\mathrm{ran\,}\Gamma _1={{\mathfrak {L}}}_1$ it suffices to prove that $\mathrm{ran\,}\Gamma _1$ is dense in ${{\mathfrak {L}}}_1$. For this assume that $v_1\perp \mathrm{ran\,}\Gamma _1$. Let ${{\widehat{g}} }=\{0,0\}\in {{\mathfrak {H}}}^2$ and ${{\widehat{v}} }=\{v_1,0\}\in {{\mathfrak {L}}}$. Then $\{{{\widehat{g}} },{{\widehat{v}} }\}$ satisfies the identity (3.2) for all $\{{{\widehat{f}} },{{\widehat{u}} }\}\in \Gamma$, and hence assumption (2) in Definition 3.1 implies that $\{{{\widehat{g}} },{{\widehat{v}} }\}\in \Gamma$. This means that ${{\widehat{v}} }\in \mathrm{mul\,}\Gamma$ and then, in particular, $v_1\in \mathrm{mul\,}\Gamma _1\subseteq \mathrm{ran\,}\Gamma _1$. Thus, $v_1=0$ and this proves that $\mathrm{ran\,}\Gamma _1={{\mathfrak {L}}}_1$. The equality $\mathrm{ran\,}\Gamma _2={{\mathfrak {L}}}_2$ is then clear by symmetry.

• (iii) The identities $\mathrm{mul\,}\Gamma _j=P_{{{\mathfrak {L}}}_j}(\mathrm{mul\,}\Gamma )$, $j=1,2$, are clear from the definition of $\Gamma$. Hence, $\mathrm{mul\,}\Gamma =\{0\}$ implies that $\mathrm{mul\,}\Gamma _1=\mathrm{mul\,}\Gamma _2=\{0\}$. Conversely, assume that e.g. $\mathrm{mul\,}\Gamma _1=\{0\}$ and that ${{\widehat{v}} }\in \mathrm{mul\,}\Gamma$. Then ${{\widehat{v}} }=\{0,v_2\}$ and hence for all $\{{{\widehat{f}} },{{\widehat{u}} }\}\in \Gamma$ the identity (3.2) implies that

  $$\begin{aligned} 0=(u_1,0)_{{{\mathfrak {L}}}_1}-(u_2,v_2)_{{{\mathfrak {L}}}_2}=-(u_2,v_2)_{{{\mathfrak {L}}}_2}. \end{aligned}$$

  In item (ii) it was shows that $\mathrm{ran\,}\Gamma _2={{\mathfrak {L}}}_2$ and, thus, one concludes that $v_2=0$. Hence, $\mathrm{mul\,}\Gamma =\{0\}$. Similarly, $\mathrm{mul\,}\Gamma _2=\{0\}$ implies $\mathrm{mul\,}\Gamma =\{0\}$.

• (iv) Since $V_j={\mathrm{ker\,}\,}\Gamma _j$ and $\Gamma _j$ is closed by item (i) also $V_j$ is closed $j=1,2$. If ${{\widehat{f}} }\in V_1$ and ${{\widehat{g}} }\in V_2$ then it follows from (3.2) that $[{{\widehat{f}} },{{\widehat{g}} } ]_{{{\mathfrak {H}}}^2}=0$ and in view of (3.1) this means that the inclusions $V_1\subseteq V_2^{-[*]}$ and $V_2\subseteq V_1^{-[*]}$ hold; these inclusions are clearly equivalent to each other.

• (v) The definition in (3.6) shows that

  $$\begin{aligned} \mathrm{mul\,}V_2=P_{{\mathfrak {H}}}(\mathrm{mul\,}{{\mathcal {U}}}). \end{aligned}$$

  Hence, $\mathrm{mul\,}{{\mathcal {U}}}=\{0\}$ implies $\mathrm{mul\,}V_2=\{0\}$. Conversely, if $\{f',u_1\}\in \mathrm{mul\,}{{\mathcal {U}}}$ then $f'\in \mathrm{mul\,}V_2$ and if $\mathrm{mul\,}V_2=\{0\}$ then $f'=0$. Now (3.2) implies that $(u_1,u_1)_{{{\mathfrak {L}}}_1}=0$ and thus also $u_1=0$, i.e., $\mathrm{mul\,}{{\mathcal {U}}}=\{0\}$. The equivalence of ${\mathrm{ker\,}\,}{{\mathcal {U}}}=\{0\}$ and ${\mathrm{ker\,}\,}V_1=\{0\}$ can be seen in the same way. As to the last equivalence notice that $\kappa _-({{\mathfrak {H}}}\times {{\mathfrak {L}}}_1)=\kappa _-({{\mathfrak {H}}})=\kappa _-({{\mathfrak {H}}}\times {{\mathfrak {L}}}_2)$. Now, according to item (v) in Lemma 2.5${\mathrm{ker\,}\,}{{\mathcal {U}}}=\{0\}$ is equivalent to $\mathrm{mul\,}{{\mathcal {U}}}=\{0\}$ and in this case ${{\mathcal {U}}}$ is a standard unitary operator. $\square$

Later it is shown that the inclusions in (iv) of Proposition 3.4 actually hold as equalities; see Theorem 3.15. In the special case that ${{\mathfrak {H}}}$ is a Hilbert space and V is an isometry in ${{\mathfrak {H}}}$ Proposition 3.4 can be specialized as follow.

Corollary 3.5

Let $({{{\mathfrak {L}}}},\Gamma )$ be a unitary boundary pair for an isometric operator V in the Hilbert space ${{\mathfrak {H}}}$. Then the properties (i)–(iv) in Proposition 3.4 hold and, moreover, $V_2$ and $V_1^{-1}$ are contractive operators. Furthermore, the main transform ${{\mathcal {U}}}$ from ${{\mathfrak {H}}}\times {{\mathfrak {L}}}_2$ to ${{\mathfrak {H}}}\times {{\mathfrak {L}}}_1$ of $\Gamma$ is a standard Hilbert space unitary operator.

Proof

The fact that in the Hilbert space case $V_2$ and $V_1^{-1}$ are contractive operators follows from (3.2) which with the choice ${{\widehat{f}} }={{\widehat{g}} }$ and ${{\widehat{u}} }={{\widehat{v}} }$ can be rewritten as

$$\begin{aligned} \Vert f\Vert _{{\mathfrak {H}}}^2-\Vert f'\Vert _{{\mathfrak {H}}}^2=\Vert u_1\Vert _{{{\mathfrak {L}}}_1}^2-\Vert u_2\Vert _{{{\mathfrak {L}}}_2}^2. \end{aligned}$$

In particular, with $u_1=0$ the condition $f'=0$ implies $f=0$ and hence ${\mathrm{ker\,}\,}V_1=\{0\}$. Similarly with $u_2=0$ the condition $f=0$ implies $f'=0$ and thus $\mathrm{mul\,}V_2=\{0\}$. According to item (v) in Proposition 3.4 one has $\mathrm{mul\,}{{\mathcal {U}}}=\{0\}$ and thus ${{\mathcal {U}}}\in {\mathbf {B}}({{\mathfrak {H}}})$ is a standard unitary operator. $\square$

3.2 The Weyl Function and the $\gamma$-fields of a Unitary Boundary Pair

Define the set ${{\mathfrak {N}}}_\lambda (V_*)$ as the intersection of ${{\mathfrak {N}}}_\lambda$ and $V_*$:

$$\begin{aligned} {{\mathfrak {N}}}_\lambda (V_*):=\left\{ f_\lambda :\begin{pmatrix}f_\lambda \\ \lambda f_\lambda \end{pmatrix}\in V_*\right\} \,\left( \subseteq {{\mathfrak {N}}}_\lambda =\left\{ f_\lambda :\begin{pmatrix}f_\lambda \\ \lambda f_\lambda \end{pmatrix}\in V^{-{[*]}}\right\} \right) \end{aligned}$$

and the corresponding subset of $V_*$:

$$\begin{aligned} {{\widehat{{\mathfrak {N}}}} }_\lambda (V_*)=\left\{ {{\widehat{f}_\lambda } }=\begin{pmatrix}f_\lambda \\ \lambda f_\lambda \end{pmatrix}:{{\widehat{f}_\lambda } }\in V_*\right\} . \end{aligned}$$

If ${\widehat{f}}_{\lambda }\in \widehat{{{\mathfrak {N}}}}_{\lambda }(V_*)$ and ${\widehat{f}}_{\mu }\in \widehat{{{\mathfrak {N}}}}_{\mu }(V_*)$ then for some ${{\widehat{u}}},{{\widehat{v}}}\in {{{\mathfrak {L}}}}$ one has

$$\begin{aligned} \left\{ {\widehat{f}}_\lambda , \begin{pmatrix} u_1 \\ u_2 \end{pmatrix}\right\} \in \Gamma , \quad \left\{ {\widehat{f}}_\mu , \begin{pmatrix} v_1 \\ v_2 \end{pmatrix}\right\} \in \Gamma \end{aligned}$$

and an application of (3.2) shows that

$$\begin{aligned} (1-\lambda \overline{\mu } )[f_{\lambda },f_{\mu }]_{{{\mathfrak {H}}}}=(u_1,v_1)_{{{\mathfrak {L}}}_1}-(u_2,v_2)_{{{\mathfrak {L}}}_2}. \end{aligned}$$

(3.9)

Lemma 3.6

(cf. [16, Lemma 3.2]) Let $u_j$ belong to a Hilbert space ${{{\mathcal {H}}}}$, let $\lambda _j\in {\mathbb {C}}$, and define the vector valued function $f_j$ by

$$\begin{aligned} f_j(\lambda ):=\frac{u_j}{1-\lambda _j\lambda }, \quad j=1,\dots ,n. \end{aligned}$$

Then the Gram matrix of the functions $f_j(\lambda )$ $(j=1,\dots ,n)$ in the space $L_2({{{\mathcal {H}}}})$ over the unit circle

• (i) is equal to $G=\left( \frac{(u_j,u_k)_{{{{\mathcal {H}}}}}}{1-\lambda _j\overline{\lambda }_k}\right) _{j,k=1}^n$, if $\lambda _j\in {{\mathbb {D}}}$ $(j,k=1,\dots ,n)$;

• (ii) is equal to $-G=\left( -\frac{(u_j,u_k)_{{{{\mathcal {H}}}}}}{1-\lambda _j\overline{\lambda }_k}\right) _{j,k=1}^n$, if $\lambda _j\in {{\mathbb {D}}}_e$ $(j,k=1,\dots ,n)$.

Proof

To determine the Gram matrix consider the inner product of the ${{\mathcal {H}}}$-valued functions $f_j(\lambda )$ and $f_k(\lambda )$ with $\lambda =e^{it}$, $t\in [0,2\pi ]$, for $j,k=1,\dots ,n$.

• (i) If $\lambda _j\in {{\mathbb {D}}}$ then in view of the equalities

  $$\begin{aligned} \begin{aligned} \left( f_j,f_k\right) _{L_2({{{\mathcal {H}}}})}&=\frac{1}{2\pi }\int \limits _0^{2\pi } \frac{dt}{(1-\lambda _je^{it})(1-\overline{\lambda }_k e^{-it})}(u_j,u_k)_{{{{\mathcal {H}}}}}\\&=\frac{1}{2\pi i}\oint \frac{d\lambda }{(1-\lambda _j\lambda )(\lambda -\overline{\lambda }_k)}(u_j,u_k)_{{{{\mathcal {H}}}}}=\frac{(u_j,u_k)_{{{{\mathcal {H}}}}}}{{1-\lambda _j\overline{\lambda }_k}} \end{aligned} \end{aligned}$$

  the matrix G coincides with the Gram matrix of the functions $f_j(\lambda )$ $(j=1,\dots ,n)$ in the space $L_2({{{\mathcal {H}}}})$ on the unit circle.

• (ii) Analogously, if $\lambda _j\in {{\mathbb {D}}}_e$ $(j=1,\dots ,n)$ then it follows from

  $$\begin{aligned} \frac{1}{2\pi i}\oint \frac{d\lambda }{(1-\lambda _j\lambda )(\lambda -\overline{\lambda }_k)}=\frac{1}{2\pi i}\oint \frac{d\lambda }{-\lambda _j(\lambda -1/\lambda _j)(\lambda -\overline{\lambda }_k)}=-\frac{1}{{1-\lambda _j\overline{\lambda }_k}} \end{aligned}$$

  that the matrix G=$\left( \frac{(u_j,u_k)_{{{{\mathcal {H}}}}}}{1-\lambda _j\overline{\lambda }_k}\right) _{j,k=1}^n$ differs in sign from the Gram matrix for functions $f_j(\lambda )$, $j=1,\dots ,n$. $\square$

Proposition 3.7

Let V be an isometric operator in a Pontryagin space ${{{\mathfrak {H}}}}$ and let $({{{\mathfrak {L}}}},\Gamma )$ be a unitary boundary pair for V. Then:

• (i) the set $(\sigma _p(V_2)\setminus \sigma _p(V))\cap {{\mathbb {D}}}_e$ consists of at most $\kappa$ points and the corresponding eigenspaces are negative;

• (ii) the set $(\sigma _p(V_1)\setminus \sigma _p(V))\cap {{\mathbb {D}}}$ consists of at most $\kappa$ points and the corresponding eigenspaces are negative.

Proof

• (i) A point $\lambda \in {{\mathbb {D}}}_e$ belongs to the set $\sigma _p(V_2)$ if and only if there exists ${\widehat{f}}_{\lambda }\in \widehat{{{\mathfrak {N}}}}_{\lambda }(V_*)$, ${f}_{\lambda }\ne 0$, such that $\left\{ {\widehat{f}}_\lambda , \begin{pmatrix} u_1 \\ 0 \end{pmatrix}\right\} \in \Gamma$ for some $u_1\in {{{\mathfrak {L}}}}_1$. The assumption $\lambda \not \in \sigma _p(V)$ means that $u_1\ne 0$.

  Now assume that $\lambda _1,\dots ,\lambda _{\kappa +1}\in {\mathbb {D}}_e$ and for some linearly independent vectors ${\widehat{f}}_{\lambda _j}\in \widehat{{{\mathfrak {N}}}}_{\lambda _j}(V_*)\setminus \{0\}$ and some $u_{1,j}\in {{{\mathfrak {L}}}}_1\setminus \{0\}$,

  $$\begin{aligned} \left\{ {\widehat{f}}_{\lambda _j}, \begin{pmatrix} u_{1,j} \\ 0 \end{pmatrix} \right\} \in \Gamma ,\quad j=1,\dots ,\kappa +1. \end{aligned}$$

  Then from (3.9) one gets

  $$\begin{aligned}{}[f_{\lambda _j},f_{\lambda _k}]_{{{\mathfrak {H}}}}=\frac{(u_{1,j},u_{1,k})_{{{\mathfrak {L}}}_1}}{{1-\lambda _j\overline{\lambda }_k}}, \quad j,k=1,\dots ,\kappa +1. \end{aligned}$$

  If $\lambda _j=\lambda _k$ for some $j\ne k$ then the vectors $u_j$ and $u_k$ are linearly independent by the assumptions $\lambda _j\not \in \sigma _p(V)$. On the other hand, if $\lambda _j\ne \lambda _k$, then the vector functions $f_j(\lambda )$ and $f_k(\lambda )$ defined in Lemma 3.6 are also linearly independent. Hence the matrix G in Lemma 3.6 is invertible. One concludes that the form

  $$\begin{aligned} \sum \limits _{j,k=1}^{\kappa +1}[f_{\lambda _j},f_{\lambda _k}]_{{{\mathfrak {H}}}}\xi _j\overline{\xi }_k= \left[ \sum \limits _{j=1}^{\kappa +1}\xi _jf_{\lambda _j},\sum \limits _{k=1}^{\kappa +1}\xi _kf_{\lambda _k}\right] _{{{\mathfrak {H}}}} \end{aligned}$$

  is negative for linearly independent vectors ${\widehat{f}}_{\lambda _j}$ when $\lambda _j\in {\mathbb {D}}_e$. This contradicts the assumption that the Pontryagin space ${{{\mathfrak {H}}}}$ has negative index $\kappa$.

• (ii) The second statement is proved analogously. $\square$

In the sequel the following two subsets of ${{\mathbb {D}}}$ and ${{\mathbb {D}}}_e$ (cf. (2.10)) will often appear:

$$\begin{aligned} {{\mathcal {D}}}:={{\mathbb {D}}}\backslash \sigma _p(V_1),\quad {{\mathcal {D}}}_e:={{\mathbb {D}}}_e\backslash \sigma _p(V_2). \end{aligned}$$

(3.10)

It should be noted that for various realization results and for the study of proper extensions of the isometry V it is typically sufficient to assume that $\sigma _p(V)=\emptyset$; this is the case in particular when V is a simple isometric operator in ${{\mathfrak {H}}}$. In this case Proposition 3.7 shows that both of the sets $\sigma _p(V_2)\cap {{\mathbb {D}}}_e$ and $\sigma _p(V_1)\cap {{\mathbb {D}}}$ contain at most $\kappa$ points.

Now consider the restrictions of (the graphs of) $\Gamma _1$ and $\Gamma _2$ to ${{\widehat{{\mathfrak {N}}}} }_\lambda (V_*)$,

$$\begin{aligned} \Gamma _1\upharpoonright {{\widehat{{\mathfrak {N}}}} }_\lambda (V_*):=\left\{ \left\{ {{\widehat{f}} },\begin{pmatrix} u_1 \\ 0 \end{pmatrix}\right\} \in \Gamma _1: {{\widehat{f}} }\in {{\widehat{{\mathfrak {N}}}} }_\lambda (V_*),\, u_1\in {{\mathfrak {L}}}_1 \right\} \end{aligned}$$

and

$$\begin{aligned} \Gamma _2\upharpoonright {{\widehat{{\mathfrak {N}}}} }_\lambda (V_*):=\left\{ \left\{ {{\widehat{f}} },\begin{pmatrix} 0 \\ u_2 \end{pmatrix}\right\} \in \Gamma _2: {{\widehat{f}} }\in {{\widehat{{\mathfrak {N}}}} }_\lambda (V_*),\, u_2\in {{\mathfrak {L}}}_2 \right\} . \end{aligned}$$

It follows from (3.10) that

$$\begin{aligned} {\mathrm{ker\,}\,}\left( \Gamma _1\upharpoonright {{\widehat{{\mathfrak {N}}}} }_\lambda (V_*)\right) =\{0\} \quad (\lambda \in {{\mathcal {D}}});\quad {\mathrm{ker\,}\,}\left( \Gamma _2\upharpoonright {{\widehat{{\mathfrak {N}}}} }_\lambda (V_*)\right) =\{0\} \quad (\lambda \in {{\mathcal {D}}}_e) \end{aligned}$$

and the assumption $\sigma _p(V)=\emptyset$ guarantees in particular that the inverses

$$\begin{aligned} {{\widehat{\gamma }} }_1(\lambda )=\left(\Gamma _1\upharpoonright {{\widehat{{\mathfrak {N}}}} }_\lambda (V_*)\right)^{-1}\; (\lambda \in {{\mathcal {D}}});\quad {{\widehat{\gamma }} }_2(\lambda )=\left(\Gamma _2\upharpoonright {{\widehat{{\mathfrak {N}}}} }_\lambda (V_*)\right)^{-1}\; (\lambda \in {{\mathcal {D}}}_e)\nonumber \\ \end{aligned}$$

(3.11)

determine single-valued operator functions, which will be denoted by the same symbols

$$\begin{aligned} {{\widehat{\gamma }} }_1(\lambda ):{{{\mathfrak {L}}}}_1\rightarrow {{\widehat{{\mathfrak {N}}}} }_\lambda (V_*) \quad (\lambda \in {{\mathcal {D}}});\qquad {{\widehat{\gamma }} }_2(\lambda ):{{{\mathfrak {L}}}}_2\rightarrow {{\widehat{{\mathfrak {N}}}} }_\lambda (V_*)\quad (\lambda \in {{\mathcal {D}}}_e). \end{aligned}$$

Let $\pi _1$ and $\pi _2$ be projections onto ${{\mathfrak {L}}}_1$ and ${{\mathfrak {L}}}_2$ in ${{\mathfrak {L}}}$, respectively.

Definition 3.8

The operator functions

$$\begin{aligned} \gamma _1(\lambda )=\pi _1{{\widehat{\gamma }} }_1(\lambda )\quad (\lambda \in {{\mathcal {D}}}) \quad and \quad \gamma _2(\lambda )=\pi _1{{\widehat{\gamma }} }_2(\lambda )\quad (\lambda \in {{\mathcal {D}}}_e) \end{aligned}$$

(3.12)

will be called the $\gamma$-fields of the unitary boundary pair $({{{\mathfrak {L}}}},\Gamma )$.

The definition of the $\gamma$-fields of the unitary boundary pair $({{{\mathfrak {L}}}},\Gamma )$ yields the following explicit formulas:

$$\begin{aligned}&\left\{ \begin{pmatrix} f_\lambda \\ \lambda f_\lambda \end{pmatrix},\begin{pmatrix} u_1 \\ u_2 \end{pmatrix}\right\} \in \Gamma , \quad {{\widehat{f}} }_\lambda \in {{\widehat{{\mathfrak {N}}}} }_\lambda (V_*),\,\; \lambda \in {{\mathcal {D}}}\quad \Longrightarrow \quad \gamma _1(\lambda )u_1= f_\lambda \nonumber \\ \end{aligned}$$

(3.13)

$$\begin{aligned}&\left\{ \begin{pmatrix} f_\lambda \\ \lambda f_\lambda \end{pmatrix},\begin{pmatrix} u_1 \\ u_2 \end{pmatrix}\right\} \in \Gamma , \quad {{\widehat{f}} }_\lambda \in {{\widehat{{\mathfrak {N}}}} }_\lambda (V_*),\,\; \lambda \in {{\mathcal {D}}}_e\quad \Longrightarrow \quad \gamma _2(\lambda )u_2= f_\lambda .\nonumber \\\nonumber \\ \end{aligned}$$

(3.14)

Later it is shown that $\gamma _1(\lambda )$ and $\gamma _2(\lambda )$ are bounded everywhere defined operators and holomorphic in $\lambda$; see Theorem 3.15.

Definition 3.9

The family of linear relations defined by

$$\begin{aligned} \Theta (\lambda )= \left\{ \begin{pmatrix} u_1 \\ u_2 \end{pmatrix}:\, \left\{ {{\widehat{f}} }_\lambda ,\begin{pmatrix} u_1 \\ u_2 \end{pmatrix}\right\} \in \Gamma ,\,{{\widehat{f}} }_\lambda \in {{\widehat{{\mathfrak {N}}}} }_\lambda (V_*)\right\} ,\quad \lambda \in {{\mathcal {D}}}, \end{aligned}$$

(3.15)

will be called the Weyl family of V corresponding to the unitary boundary pair $({{{\mathfrak {L}}}},\Gamma )$, or, briefly, the Weyl family of the unitary boundary pair $({{{\mathfrak {L}}}},\Gamma )$.

In Theorem 3.15 it will be shown that the formula (3.15) determines a single-valued operator function, which is called the Weyl function of V corresponding to the boundary triple $({{\mathfrak {L}}},\Gamma )$. If the map** $\Gamma$ is single-valued, then the Weyl function $\Theta (\lambda )$ can be defined by using the $\gamma$-fields

$$\begin{aligned} \Theta (\lambda )=\Gamma _2{{\widehat{\gamma }} }_1(\lambda ), \quad \lambda \in {{\mathcal {D}}}. \end{aligned}$$

(3.16)

3.3 Unitary Boundary Pairs and Unitary Colligations

In the present section we consider a unitary boundary pair whose main transform is a unitary colligation

$$\begin{aligned} \Delta =({{{\mathfrak {H}}}},{{{\mathfrak {L}}}}_1,{{{\mathfrak {L}}}}_2, U),\quad U=\left( \begin{array}{cc} T &{} F \\ G &{} H \\ \end{array} \right) :\begin{pmatrix} {{{\mathfrak {H}}}} \\ {{{\mathfrak {L}}}}_1 \end{pmatrix}\rightarrow \begin{pmatrix} {{{\mathfrak {H}}}} \\ {{{\mathfrak {L}}}}_2 \end{pmatrix} \end{aligned}$$

(3.17)

and write explicit formulas for all the objects connected with this unitary boundary pair in terms of the blocks of U.

Theorem 3.10

Let $\Delta =({{{\mathfrak {H}}}},{{{\mathfrak {L}}}}_1,{{{\mathfrak {L}}}}_2, U)$ be a unitary colligation of the form (3.17), let ${{\mathcal {U}}}=\text {gr }U^{[*]}$, let $\Gamma ={{\mathcal {J}}}^{-1}({{\mathcal {U}}})$ be a unitary relation from $({{{\mathfrak {H}}}}^2, J_{{{{\mathfrak {H}}}}^2})$ to $({{{\mathfrak {L}}}}, J_{{{{\mathfrak {L}}}}})$, i.e. ${{\mathcal {U}}}={{\mathcal {J}}}(\Gamma )$ as in Lemma 3.3, and let

$$\begin{aligned} V:={\mathrm{ker\,}\,}\Gamma ,\quad V_*:=\mathrm{dom\,}\Gamma ,\quad V_1:={\mathrm{ker\,}\,}\Gamma _1,\quad V_2:={\mathrm{ker\,}\,}\Gamma _2. \end{aligned}$$

Then the following statements hold:

• (i) The pair $({{{\mathfrak {L}}}},\Gamma )$ is a unitary boundary pair for V.

• (ii) The unitary relation $\Gamma$ admits the representations

  $$\begin{aligned} \begin{aligned} \Gamma&=\left\{ \left\{ \begin{pmatrix} Th+Fu_1\\ h \end{pmatrix}, \begin{pmatrix} u_1\\ Gh+Hu_1\end{pmatrix}\right\} : \begin{array}{c} h\in {{\mathfrak {H}}}\\ u_1\in {{{{\mathfrak {L}}}}_1} \end{array}\right\} \\&=\left\{ \left\{ \begin{pmatrix} g\\ T^{[*]}g+G^{[*]}u_2 \end{pmatrix}, \begin{pmatrix} F^{[*]}g+H^*u_2\\ u_2\end{pmatrix}\right\} :\, \begin{array}{c} g\in {{\mathfrak {H}}}\\ u_2\in {{{{\mathfrak {L}}}}_2} \end{array}\right\} . \end{aligned} \end{aligned}$$

  (3.18)

• (iii) The isometry V in $({{{\mathfrak {H}}}},J_{{\mathfrak {H}}})$ admits the representations

  $$\begin{aligned} V^{-1}=T\upharpoonright {\mathrm{ker\,}\,}G,\quad V=T^{[*]}\upharpoonright {\mathrm{ker\,}\,}F^{[*]} \end{aligned}$$

  (3.19)

  and the linear relation $V_*$ takes the form

  $$\begin{aligned} V_*=\left\{ \begin{pmatrix} Th+Fu_1\\ h\end{pmatrix}:\, \begin{array}{c} h\in {{\mathfrak {H}}}\\ u_1\in {{{{\mathfrak {L}}}}_1} \end{array} \right\} = \left\{ \begin{pmatrix} g\\ T^{[*]}g+G^{[*]}u_2 \end{pmatrix}:\, \begin{array}{c} g\in {{\mathfrak {H}}}\\ u_2\in {{{{\mathfrak {L}}}}_2} \end{array} \right\} .\nonumber \\ \end{aligned}$$

  (3.20)

• (iv) The multivalued part of $\Gamma$ has the representations

  $$\begin{aligned} \mathrm{mul\,}\Gamma =\left\{ \begin{pmatrix} u_1\\ Hu_1 \end{pmatrix}: u_1\in {\mathrm{ker\,}\,}F \right\} =\left\{ \begin{pmatrix} H^*u_2\\ u_2 \end{pmatrix}: v\in {\mathrm{ker\,}\,}G^{[*]} \right\} . \end{aligned}$$

  (3.21)

• (v) The linear relations $V_1$ and $V_2$ are given by

  $$\begin{aligned} V_1=\left\{ \begin{pmatrix} Th \\ h \end{pmatrix}: h\in {{{\mathfrak {H}}}}\right\} , \quad V_2=\left\{ \begin{pmatrix} g \\ T^{[*]}g \end{pmatrix}: g\in {{{\mathfrak {H}}}}\right\} , \end{aligned}$$

  (3.22)

  and hence the sets ${{\mathcal {D}}}:=\rho (V_1)\cap {{\mathbb {D}}}$ and ${{\mathcal {D}}}_e:=\rho (V_2)\cap {{\mathbb {D}}}_e$ are nonempty, they coincide with the sets in (3.10) and are connected by

  $$\begin{aligned} {{\mathcal {D}}}_e={{\mathcal {D}}}^\circ :=\{\lambda \in {{\mathbb {D}}}_e:{1}/{\overline{\lambda }}\in {{\mathcal {D}}}\}, \end{aligned}$$

  (3.23)

  moreover, $\mathrm{card\,}({{\mathbb {D}}}\setminus {{\mathcal {D}}})=\mathrm{card\,}({{\mathbb {D}}}_e\setminus {{\mathcal {D}}}_e)\le \kappa$.

Proof

• (i) By Lemma 3.3$\Gamma ={{\mathcal {J}}}^{-1}{{\mathcal {U}}}$ is a unitary relation from $({{{\mathfrak {H}}}}^2, J_{{{{\mathfrak {H}}}}^2})$ to $({{{\mathfrak {L}}}}, J_{{{{\mathfrak {L}}}}})$. This means that the assumptions (1), (2) of Definition 3.1 are satisfied and, therefore, $({{{\mathfrak {L}}}},\Gamma )$ is a unitary boundary pair for the isometry $V={\mathrm{ker\,}\,}\Gamma$ (which is an operator in view of (3) below).

• (ii) Since the operator $U:\begin{pmatrix} {{\mathfrak {H}}}\\ {{\mathfrak {L}}}_1 \end{pmatrix}\rightarrow \begin{pmatrix} {{\mathfrak {H}}}\\ {{\mathfrak {L}}}_2 \end{pmatrix}$ is unitary then ${\mathcal {U}}=\mathrm{gr\,}U^{[*]}$ is also the graph of the operator $U^{-1}$ in (3.17), and hence ${\mathcal {U}}$ has the following representations

  $$\begin{aligned} {\mathcal {U}}=\mathrm{gr\,}U^{-1}=\left\{ \left\{ \begin{pmatrix} Th+Fu_1\\ Gh+Hu_1 \end{pmatrix}, \begin{pmatrix} h\\ u_1 \end{pmatrix}\right\} :\,\begin{array}{c} h\in {{\mathfrak {H}}}\\ u_1\in {{\mathfrak {L}}}_1 \end{array}\right\} . \end{aligned}$$

  In view of (3.6) this yields the first formula in (3.18). The equality

  $$\begin{aligned} {\mathcal {U}}=\text {gr }{U^{[*]}}=\left\{ \left\{ \begin{pmatrix} g\\ u_2 \end{pmatrix}, \begin{pmatrix} T^{[*]}g+G^{[*]}u_2\\ F^{[*]}g+H^{[*]}u_2 \end{pmatrix}\right\} :\,\begin{array}{c} g\in {{\mathfrak {H}}}\\ u_2\in {{\mathfrak {L}}}_2 \end{array}\right\} . \end{aligned}$$

  leads to the second representation of $\Gamma$ in (3.18).

• (iii) and (iv) The formulas (3.19), (3.20), and (3.21) are all implied by (3.18).

• (v) The formulas (3.22) for $V_1={\mathrm{ker\,}\,}\Gamma _1$ and $V_2={\mathrm{ker\,}\,}\Gamma _2$ are again obtained from (3.18). In particular, $V_2$ is the graph of the bounded operator $T^{[*]}$. It is closed and ${{\mathcal {D}}}_e=\rho (V_2)=\rho (T^{[*]})\ne \emptyset$. Since $T^{[*]}$ is a contractive operator in the Pontryagin space $({{\mathfrak {H}}}, J_{{\mathfrak {H}}})$, its spectrum in ${\mathbb {D}}_e$ consists of at most $\kappa$ eigenvalues, so

  $$\begin{aligned} \mathrm{card\,}({{\mathbb {D}}}_e\backslash {{\mathcal {D}}}_e)=\mathrm{card\,}({{\mathbb {D}}}_e\backslash \rho (T^{[*]}))\le \kappa , \end{aligned}$$

  see e.g. [31, p.91 Lemma 11.8]. Similarly, $V_1^{-1}$ is the graph of the contractive operator T in the Pontryagin space $({{\mathfrak {H}}}, J_{{\mathfrak {H}}})$. Therefore,

  $$\begin{aligned} {{\mathcal {D}}}=\rho (V_1)\cap {{\mathbb {D}}}=\rho (T^{-1})\cap {{\mathbb {D}}}=(\rho (T^{[*]})\cap {{\mathbb {D}}}_e)^\circ ={{\mathcal {D}}}_e^\circ . \end{aligned}$$

  This completes the proof. $\square$

By Proposition 3.2 the closure of $V_*$ is $V^{-[*]}$. Hence, by taking closures in (3.20) one arrives at the following representations for $V^{-[*]}$:

$$\begin{aligned} V^{-[*]}=\left\{ \begin{pmatrix} Th+f\\ h \end{pmatrix}:\,\begin{array}{c} h\in {{\mathfrak {H}}}\\ f\in {{\mathfrak {H}}}[-]{\mathrm{ker\,}\,}F^{[*]} \end{array}\right\} , \end{aligned}$$

(3.24)

and

$$\begin{aligned} V^{-[*]}=\left\{ \begin{pmatrix} g\\ T^{[*]}g+f \end{pmatrix}:\,\begin{array}{c} g\in {{\mathfrak {H}}}\\ f\in {{\mathfrak {H}}}[-]{\mathrm{ker\,}\,}G \end{array}\right\} . \end{aligned}$$

(3.25)

Theorem 3.11

Let $\Delta =({{{\mathfrak {H}}}},{{{\mathfrak {L}}}}_1,{{{\mathfrak {L}}}}_2, U)$ be a unitary colligation of the form (3.17), let ${{\mathcal {U}}}=\text {gr }U^{[*]}$, let $\Gamma ={{\mathcal {J}}}^{-1}U$ and let the corresponding Weyl function $\Theta (\lambda )$ of V and the $\gamma$-fields $\gamma _1(\lambda )$ and $\gamma _2(\lambda )$ be given by (3.15) and (3.13), (3.14). Then the following statements hold:

• (i) The defect subspace ${{\mathfrak {N}}}_\lambda (V_*)$ takes the form

  $$\begin{aligned} {{\mathfrak {N}}}_\lambda (V_*)= & {} \{(I-\lambda T)^{-1}Fu_1:\,u_1 \in {{{{\mathfrak {L}}}}_1}\},\quad \lambda \in {{\mathcal {D}}}; \end{aligned}$$

  (3.26)
  $$\begin{aligned} {{\mathfrak {N}}}_\lambda (V_*)= & {} \{(\lambda I- T^{[*]})^{-1}G^{[*]}u_2:\, u_2 \in {{{{\mathfrak {L}}}}_2}\}, \quad \lambda \in {{\mathcal {D}}}_e. \end{aligned}$$

  (3.27)

• (ii) $\Gamma _1({{\widehat{{\mathfrak {N}}}} }_\lambda (V_*))={{{\mathfrak {L}}}}_1$, $\lambda \in {{\mathcal {D}}}$, and $\Gamma _2({{\widehat{{\mathfrak {N}}}} }_\lambda (V_*))={{{\mathfrak {L}}}}_2$, $\lambda \in {{\mathcal {D}}}_e$, and the $\gamma$-fields $\gamma _1(\lambda )$ and $\gamma _2(\lambda )$ take the form

  $$\begin{aligned} \gamma _1(\lambda )= & {} (I-\lambda T)^{-1}F,\quad \lambda \in {{\mathcal {D}}};\\ \gamma _2(\lambda )= & {} (\lambda I-T^{[*]})^{-1}G^{[*]}, \quad \lambda \in {{\mathcal {D}}}_e. \end{aligned}$$

• (iii) The Weyl function $\Theta (\lambda )$ of V corresponding to the unitary boundary pair $({{{\mathfrak {L}}}},\Gamma )$ coincides with the characteristic function of the colligation $\Delta$, i.e.,

  $$\begin{aligned} {\Theta }(\lambda )=\Theta _\Delta (\lambda )=H+\lambda G(I-\lambda T)^{-1}F,\quad \lambda \in {{\mathcal {D}}}. \end{aligned}$$

• (iv) The unitary colligation $\Delta$ is closely connected if and only if the operator V is simple.

Proof

• (i) Recall that ${{\widehat{{\mathfrak {N}}}} }_\lambda (V_*)$ consists of vectors $\begin{pmatrix}f_\lambda \\ \lambda f_\lambda \end{pmatrix}\in V_*$. Therefore, the vector $\begin{pmatrix} Th_\lambda +Fu_1\\ h_\lambda \end{pmatrix}$ $(h_\lambda \in {{\mathfrak {H}}},\, u_1\in {{\mathfrak {L}}}_1)$ belongs to ${{\widehat{{\mathfrak {N}}}} }_\lambda (V_*)$ precisely when

  $$\begin{aligned} h_\lambda =\lambda (Th_\lambda +Fu_1). \end{aligned}$$

  Hence, for $\lambda \in {{\mathcal {D}}}=\rho (T^{-1})\cap {{\mathbb {D}}}$ one obtains

  $$\begin{aligned} h_\lambda =\lambda (I-\lambda T)^{-1}Fu_1, \quad \lambda \in {{\mathcal {D}}}. \end{aligned}$$

  Similarly, the vector $\begin{pmatrix} g_\lambda \\ T^{[*]}g_\lambda +G^{[*]}u_2 \end{pmatrix}$ $g_\lambda \in {{\mathfrak {H}}},\, u_2\in {{\mathfrak {L}}}_2$ belongs to ${{\widehat{{\mathfrak {N}}}} }_\lambda (V_*)$ if and only if

  $$\begin{aligned} T^{[*]}g_\lambda +G^{[*]}u_2=\lambda g_\lambda . \end{aligned}$$

  Hence, for $\lambda \in {{\mathcal {D}}}_e=\rho (T^{[*]})\cap {{\mathbb {D}}}_e$ one obtains

  $$\begin{aligned} g_\lambda =(\lambda I- T^{[*]})^{-1}G^{[*]}u_2, \quad \lambda \in {{\mathcal {D}}}_e. \end{aligned}$$

• (ii) By the first formula in (3.18) one gets

  $$\begin{aligned} \left\{ \begin{pmatrix} f_\lambda \\ \lambda f_\lambda \end{pmatrix}, \begin{pmatrix} u_1\\ Hu_1+\lambda G(I-\lambda T)^{-1}Fu_1 \end{pmatrix}\right\} \in \Gamma , \quad \lambda \in {{\mathcal {D}}}, \end{aligned}$$

  (3.28)

  and in view of (3.11) and (3.12) the $\gamma$-field $\gamma _1(\lambda )$ takes the form

  $$\begin{aligned} \gamma _1(\lambda ) u_1= f_\lambda =(I-\lambda T)^{-1}Fu_1, \quad \lambda \in {{\mathcal {D}}}. \end{aligned}$$

  Similarly, by the second formula in (3.18)

  $$\begin{aligned} \left\{ \begin{pmatrix} g_\lambda \\ \lambda g_\lambda \end{pmatrix}, \begin{pmatrix} H^*u_2+F^{[*]}(\lambda I-T^{[*]})^{-1}G^{[*]}u_2 \\ u_2\end{pmatrix}\right\} \in \Gamma , \quad \lambda \in {{\mathcal {D}}}_e, \end{aligned}$$

  (3.29)

  and hence

  $$\begin{aligned} \gamma _2(\lambda ) u_2= g_\lambda =(\lambda I-T^{[*]})^{-1}G^{[*]}u_2, \quad \lambda \in {{\mathcal {D}}}_e. \end{aligned}$$

• (iii) It follows also from (3.28) that

  $$\begin{aligned} {\Theta }(\lambda )u_1=\left(H+\lambda G(I-\lambda T)^{-1}F\right)u_1={\Theta }_\Delta (\lambda )u_1,\quad \lambda \in {{\mathcal {D}}},\, u_1\in {{\mathfrak {L}}}_1. \end{aligned}$$

• (iv) Notice that in view of (3.24) and (3.25) the defect subspaces ${{\mathfrak {N}}}_\lambda$ take the form

  $$\begin{aligned} {{\mathfrak {N}}}_\lambda =\{(I-\lambda T)^{-1}f:\,f \in {{{{\mathfrak {H}}}}[-]{\mathrm{ker\,}\,}F^{[*]}}\},\quad \lambda \in {{\mathcal {D}}}; \end{aligned}$$

  (3.30)
  $$\begin{aligned} {{\mathfrak {N}}}_\lambda =\{(\lambda I- T^{[*]})^{-1}g:\, g \in {{{\mathfrak {H}}}}[-]{\mathrm{ker\,}\,}G\}, \quad \lambda \in {{\mathcal {D}}}_e. \end{aligned}$$

  (3.31)

  Comparison of (3.26), (3.27) with (3.30), (3.31) shows that the subspaces ${{\mathfrak {N}}}_\lambda (V_*)$ are dense in ${{\mathfrak {N}}}_\lambda$ for all $\lambda \in {{\mathcal {D}}}\cup {{\mathcal {D}}}_e$. Therefore, the set

  $$\begin{aligned} \overline{span }\,\{{{\mathfrak {N}}}_\lambda :\,\lambda \in {{\mathcal {D}}}\}\quad (resp. \quad \overline{span }\,\{{{\mathfrak {N}}}_\lambda :\,\lambda \in {{\mathcal {D}}}_e\}) \end{aligned}$$

  coincides with the set

  $$\begin{aligned} \overline{span }\,\{{{\mathfrak {N}}}_\lambda (V_*):\,\lambda \in {{\mathcal {D}}}\}\quad (resp. \quad \overline{span }\,\{{{\mathfrak {N}}}_\lambda (V_*):\,\lambda \in {{\mathcal {D}}}_e\}). \end{aligned}$$

  In view of the formulas (3.26), (3.27) one obtains the equalities

  $$\begin{aligned}&\overline{span }\,\{{{\mathfrak {N}}}_\lambda :\,\lambda \in {{\mathcal {D}}}\}= \overline{span }\,\{\mathrm{ran\,}(T^mF):\,m\ge 0\}, \end{aligned}$$

  (3.32)
  $$\begin{aligned}&\overline{span }\,\{{{\mathfrak {N}}}_\lambda :\,\lambda \in {{\mathcal {D}}}_e\}= \overline{span }\,\{\mathrm{ran\,}(T^{[*]})^nG^{[*]}):\,n\ge 0\}. \end{aligned}$$

  (3.33)

  By Definition 2.3 and the definition in (2.14) this proves the statement (iv). $\square$

Remark 3.12

With the assumptions of Theorem 3.11 there exist uniform limits

$$\begin{aligned} (\gamma _2(\infty ):=)\lim _{\lambda \rightarrow \infty }\gamma _2(\lambda )=0,\quad \lim _{\lambda \rightarrow \infty }\lambda \gamma _2(\lambda )=G^{[*]}. \end{aligned}$$

(3.34)

3.4 General Case

To prove the desired statements for a general unitary boundary pair some preparatory lemmas will be used.

Let $\alpha \in {{\mathbb {C}}}$ such that $|\alpha |\ne 1$ and define the transform $M^{(\alpha )}\in {\mathbf {B}}({{\mathfrak {H}}}^2)$ in the space $({{{\mathfrak {H}}}}^2, J_{{{{\mathfrak {H}}}}^2})$ by

$$\begin{aligned} M^{(\alpha )}=\frac{1}{\beta }\begin{pmatrix}I &{} -\alpha I \\ -{\bar{\alpha }} I &{} I \end{pmatrix},\quad \beta :=\sqrt{1-|\alpha |^2}. \end{aligned}$$

(3.35)

It is easy to check that $M^{(\alpha )}$ is a standard unitary operator in the Kreĭn space $({{{\mathfrak {H}}}}^2, J_{{{{\mathfrak {H}}}}^2})$. Clearly, $M^{(0)}=I$ and

$$\begin{aligned} (M^{(\alpha )})^{-1}=M^{(-\alpha )}, \quad |\alpha |\ne 1. \end{aligned}$$

(3.36)

Associated with $M^{(\alpha )}$ define a transform of the extended complex plane ${{\mathbb {C}}}\cup \{\infty \}$ by the formula

$$\begin{aligned} \mu ^{(\alpha )}(\lambda ):=\frac{\lambda -{\bar{\alpha }}}{1-\alpha \lambda }. \end{aligned}$$

Lemma 3.13

The transform $M^{(\alpha )}$ in (3.35) maps closed subspaces of the Kreĭn space $({{{\mathfrak {H}}}}^2, J_{{{{\mathfrak {H}}}}^2})$ back to closed subspaces and it satisfies

$$\begin{aligned} \left( M^{(\alpha )}({{\mathcal {S}}}^{-1})\right) =\left( M^{({\bar{\alpha }})}\left( {{\mathcal {S}}}\right) \right) ^{-1} \end{aligned}$$

(3.37)

and

$$\begin{aligned} M^{(\alpha )}\left( {{\mathcal {S}}}^{-[*]}\right) =\left( M^{(\alpha )}({{\mathcal {S}}})\right) ^{-[*]}. \end{aligned}$$

(3.38)

In particular, $M^{(\alpha )}$ maps isometric (unitary, contractive, expanding) relations ${{\mathcal {S}}}$ in ${{\mathfrak {H}}}$ back to isometric (unitary, contractive, expanding) relations $M^{(\alpha )}({{\mathcal {S}}})$ in ${{\mathfrak {H}}}$ and, moreover, for any closed linear relation ${{\mathcal {S}}}$ in ${{\mathfrak {H}}}$ the following statements hold with $1-\alpha \lambda \ne 0$:

• (i) ${\mathrm{ker\,}\,}({{\mathcal {S}}}-\lambda )={\mathrm{ker\,}\,}(M^{(\alpha )}({{\mathcal {S}}})-\mu ^{(\alpha )}(\lambda )I)$, $\mathrm{mul\,}{{\mathcal {S}}}= {\mathrm{ker\,}\,}(\alpha M^{(\alpha )}({{\mathcal {S}}})+I)$;

• (ii) $\mathrm{ran\,}({{\mathcal {S}}}-\lambda )=\mathrm{ran\,}(M^{(\alpha )}({{\mathcal {S}}})-\mu ^{(\alpha )}(\lambda )I)$, $\mathrm{dom\,}{{\mathcal {S}}}= \mathrm{ran\,}(\alpha M^{(\alpha )}({{\mathcal {S}}})+I)$;

• (iii) $\lambda \in \sigma _j({{\mathcal {S}}})$ $\Longleftrightarrow$ $\mu ^{(\alpha )}(\lambda )\in \sigma _j(M^{(\alpha )}({{\mathcal {S}}}))$ for $j=p,c,r$;

• (iv) $\lambda \in \rho ({{\mathcal {S}}})$ $\Longleftrightarrow$ $\mu ^{(\alpha )}(\lambda )\in \rho (M^{(\alpha )}({{\mathcal {S}}}))$.

Proof

The map** $M^{(\alpha )}$ is unitary in the Kreĭn space $({{{\mathfrak {H}}}}^2, J_{{{{\mathfrak {H}}}}^2})$ and, in fact, also unitary and selfadjoint as a linear operator on the Hilbert space ${{\mathfrak {H}}}^2$. Therefore, it maps closed subsets to closed subsets in ${{\mathfrak {H}}}^2$. The formula (3.37) is checked with a straightforward calculation. Moreover, for all ${{\widehat{f}} }=\{f,f'\},{{\widehat{g}} }=\{g,g'\}$

$$\begin{aligned} \left( J_{{{\mathfrak {H}}}^2}{{\widehat{f}} },{{\widehat{g}} }\right) =\left( J_{{{\mathfrak {H}}}^2}M^{(\alpha )}{{\widehat{f}} },M^{(\alpha )}{{\widehat{g}} }\right) , \end{aligned}$$

(3.39)

which shows that if ${{\mathcal {S}}}$ is isometric (contractive, expanding) relation in ${{\mathfrak {H}}}$ so is its image $M^{(\alpha )}({{\mathcal {S}}})$, since the expression in (3.39) with ${{\widehat{f}} }={{\widehat{g}} }\in {{\mathcal {S}}}$ takes the value $=0$ (resp. $\ge 0$ and $\le 0$). Moreover, (3.39) implies the property (3.38).

The relation ${{\mathcal {S}}}$ in the Pontryagin space ${{\mathfrak {H}}}$ is unitary precisely when $S=S^{-[*]}$ and hence (3.39) implies that then also $M^{(\alpha )}({{\mathcal {S}}})=(M^{(\alpha )}({{\mathcal {S}}}))^{-[*]}$, i.e., $M^{(\alpha )}({{\mathcal {S}}})$ is a unitary relation in ${{\mathfrak {H}}}$.

To prove the remaining assertions let $\{f,f'\}\in {{\mathcal {S}}}$. Then $\{f,f'-\lambda f\}\in ({{\mathcal {S}}}-\lambda I)$ and this is equivalent to

$$\begin{aligned} \left( {\begin{array}{c}\frac{1}{\beta }(f-\alpha f')\\ \frac{\beta }{1-\alpha \lambda }(f'-\lambda f)\end{array}}\right) \in M^{\alpha }({{\mathcal {S}}})-\mu ^{(\alpha )}(\lambda ) I, \end{aligned}$$

where $\alpha \in {{\mathbb {D}}}$, $1-\alpha \lambda \ne 0$. This formula with $\lambda \in {{\mathbb {C}}}$ gives the equalities for ${\mathrm{ker\,}\,}({{\mathcal {S}}}-\lambda )$ and $\mathrm{ran\,}({{\mathcal {S}}}-\lambda )$. Analogously the choice $\lambda =\infty$ corresponds to $\mu ^{(\alpha )}(\infty )=-1/{\alpha }$ and this yields the formulas for $\mathrm{mul\,}{{\mathcal {S}}}$ and $\mathrm{dom\,}{{\mathcal {S}}}$ in (i) and (ii).

The statements (iii) and (iv) follow from (i) and (ii) when applying the definitions of the resolvent set $\rho ({{\mathcal {S}}})$ and the spectral components $\sigma _j({{\mathcal {S}}})$, $j=s,c,r$. $\square$

In the next lemma the transform $M^{(\alpha )}\in {\mathbf {B}}({{\mathfrak {H}}}^2)$ is composed with a unitary boundary pair.

Lemma 3.14

Let V be a closed isometric operator in a Pontryagin space ${{{\mathfrak {H}}}}$, let $({{{\mathfrak {L}}}},\Gamma )$ be a unitary boundary pair for V, and let $\alpha \in {{\mathbb {C}}}$, $|\alpha |\ne 1$. Then

$$\begin{aligned} V^{(\alpha )}:=M^{(\alpha )}(V) \end{aligned}$$

is also a closed isometric relation in ${{\mathfrak {H}}}$. Moreover, $\mathrm{mul\,}V^{(\alpha )}=\{0\}$ precisely when $\alpha ^{-1}\not \in \sigma _p(V)$, and in this case:

• (i) The composition

  $$\begin{aligned} \Gamma ^{(\alpha )}:=\Gamma \circ M^{(-\alpha )} \end{aligned}$$

  (3.40)

  defines a unitary boundary pair $({{{\mathfrak {L}}}},\Gamma ^{(\alpha )})$ for the isometric operator $V^{(\alpha )}$.

• (ii) The Weyl function $\Theta ^{(\alpha )}$ and the $\gamma$-fields $\gamma _1^{(\alpha )}$, $\gamma _2^{(\alpha )}$ of the unitary boundary pair $({{{\mathfrak {L}}}},\Gamma ^{(\alpha )})$ are connected to the Weyl function and the $\gamma$-fields of the unitary boundary pair $({{{\mathfrak {L}}}},\Gamma )$ by

  $$\begin{aligned}&\Theta ^{(\alpha )} \left( \mu (\alpha )\right) = \Theta (\lambda ),\quad \quad \lambda \in {{\mathcal {D}}}, \end{aligned}$$

  (3.41)
  $$\begin{aligned}&\gamma _1^{(\alpha )} \left( \mu (\alpha )\right) =\frac{1-\lambda \alpha }{\beta }\gamma _1(\lambda ),\quad \lambda \in {{\mathcal {D}}}, \end{aligned}$$

  (3.42)
  $$\begin{aligned}&\gamma _2^{(\alpha )} \left( \mu (\alpha )\right) =\frac{1-\lambda \alpha }{\beta }\gamma _2(\lambda ),\quad \lambda \in {{\mathcal {D}}}_e. \end{aligned}$$

  (3.43)

Proof

The statements concerning the linear relation $V^{(\alpha )}$ are implied by Lemma 3.13.

• (i) By definition $M^{(-\alpha )}$ is a standard unitary operator in the Kreĭn space $({{{\mathfrak {H}}}}^2, J_{{{{\mathfrak {H}}}}^2})$. Therefore, the composition $\Gamma ^{(\alpha )}=\Gamma \circ M^{(-\alpha )}$ is a unitary relation from the Kreĭn space $({{{\mathfrak {H}}}}^2, J_{{{{\mathfrak {H}}}}^2})$ to the Kreĭn space $({{{\mathfrak {L}}}}, J_{{{{\mathfrak {L}}}}})$. It follows from the equivalence

  $$\begin{aligned} \{{{\widehat{f}} },{{\widehat{u}} }\}\in \Gamma \Longleftrightarrow \{M^{(\alpha )}{{\widehat{f}} },{{\widehat{u}} }\}\in \Gamma ^{(\alpha )} \end{aligned}$$

  (3.44)

  that ${\mathrm{ker\,}\,}\Gamma ^{(\alpha )}=M^{(\alpha )}(V)$ and $\mathrm{dom\,}\Gamma ^{(\alpha )}=M^{(\alpha )}(V_*)$; see (3.36).

• (ii) By Lemma 3.13

  $$\begin{aligned} \lambda \in \sigma _p(V_*) \Leftrightarrow \mu ^{(\alpha )}(\lambda )\in \sigma _p(V_*^{(\alpha )}) \end{aligned}$$

  and hence the defect subspaces ${{\mathfrak {N}}}_\omega (V_*^{(\alpha )}):={\mathrm{ker\,}\,}( V_*^{(\alpha )} - \omega I)$ are connected with the defect subspaces ${{\mathfrak {N}}}_\lambda (V_*)$ by

  $$\begin{aligned} {{\mathfrak {N}}}_\lambda (V_*)={{\mathfrak {N}}}_{\mu ^{({\alpha })}(\lambda )}(V_*^{(\alpha )}). \end{aligned}$$

  (3.45)

  Rewriting the equivalence (3.44) for vectors $f\in {{\mathfrak {N}}}_\lambda (V_*)$ we obtain

  $$\begin{aligned} \left\{ \left( {\begin{array}{c} f\\ \lambda f\end{array}}\right) ,\left( {\begin{array}{c}u_1\\ u_2\end{array}}\right) \right\} \in \Gamma \Longleftrightarrow \left\{ \frac{1-\lambda \alpha }{\beta }\left( {\begin{array}{c} f\\ \mu ^{({\alpha })}(\lambda )f\end{array}}\right) ,\left( {\begin{array}{c}u_1\\ u_2\end{array}}\right) \right\} \in \Gamma ^{(\alpha )}. \end{aligned}$$

  (3.46)

  In view of Definition 3.8 this implies

  $$\begin{aligned} \gamma _1^{(\alpha )}(\mu ^{({\alpha })}(\lambda ))u_1= & {} \frac{1-\lambda \alpha }{\beta }\gamma _1(\lambda )u_1,\quad \lambda \in {{\mathcal {D}}};\\ \gamma _2^{(\alpha )}(\mu ^{({\alpha })}(\lambda ))u_2= & {} \frac{1-\lambda \alpha }{\beta }\gamma _2(\lambda )u_2,\quad \lambda \in {{\mathcal {D}}}_e. \end{aligned}$$

  By virtue of (3.50) this proves (ii). $\square$

The next two theorems contain a full characterization of the class of Weyl functions $\Theta (\lambda )$ of boundary pairs. In the first theorem it is shown that $\Theta (\lambda )$ belongs to the generalized Schur class ${{\mathcal {S}}}_\kappa ({{\mathfrak {L}}}_1,{{\mathfrak {L}}}_2)$.

Theorem 3.15

Let $({{\mathfrak {L}}},\Gamma )$ be a unitary boundary pair for an isometric operator V in a Pontryagin space ${{\mathfrak {H}}}$ and let $V_1={\mathrm{ker\,}\,}\Gamma _1$, $V_2={\mathrm{ker\,}\,}\Gamma _2$. Then:

• (i) $V_1$ and $V_2$ are closed linear relations which are connected by $V_1=V_2^{-[*]}$ and, moreover, the sets

  $$\begin{aligned} {{\mathcal {D}}}:=\rho (V_1)\cap {{\mathbb {D}}}\quad and \quad {{\mathcal {D}}}_e:=\rho (V_2)\cap {{\mathbb {D}}}_e \end{aligned}$$

  (3.47)

  coincide with the sets in (3.10) and they are connected by (3.23). In particular, they are nonempty and the sets ${{\mathbb {D}}}\setminus {{\mathcal {D}}}$ and ${{\mathbb {D}}}_e\setminus {{\mathcal {D}}}_e$ contain at most $\kappa$ points.

• (ii) The $\gamma$-field $\gamma _1(\lambda )$ is holomorphic on ${{\mathcal {D}}}$ with values in ${\mathbf {B}}({{\mathfrak {L}}}_1,{{\mathfrak {H}}})$.

• (iii) The $\gamma$-field $\gamma _2(\lambda )$ is holomorphic on ${{\mathcal {D}}}_e$ with values in ${\mathbf {B}}({{\mathfrak {L}}}_2,{{\mathfrak {H}}})$. If $0\in {{\mathcal {D}}}$ and $\gamma _2^\#$ is defined by

  $$\begin{aligned} \gamma _2^\#(\lambda ):= \gamma _2(\bar{\lambda }^{-1})^*,\quad \lambda \in {{\mathcal {D}}}, \end{aligned}$$

  (3.48)

  then the following uniform limits exist

  $$\begin{aligned} ( \gamma _2^\#(0):=)\lim _{\lambda \rightarrow 0}\gamma _2^\#(\lambda )=0,\quad ((\gamma _2^\#)'(0):=)\lim _{\lambda \rightarrow 0}\frac{1}{\lambda }\gamma _2^\#(\lambda )(\in {\mathbf {B}}({{\mathfrak {H}}},{{\mathfrak {L}}}_2)).\nonumber \\ \end{aligned}$$

  (3.49)

• (iv) The Weyl function $\Theta (\lambda )$ is holomorphic on ${{\mathcal {D}}}$, takes values in ${\mathbf {B}}({{\mathfrak {L}}}_1,{{\mathfrak {L}}}_2)$ for $\lambda \in {{\mathcal {D}}}$ and belongs to the class ${{\mathcal {S}}}_\kappa ({{\mathfrak {L}}}_1,{{\mathfrak {L}}}_2)$.

• (v) For all $\lambda \in {{\mathcal {D}}}$ the following relation holds

  $$\begin{aligned} \left\{ \begin{pmatrix} \gamma _1(\lambda )u_1 \\ \lambda \gamma _1(\lambda )u_1 \end{pmatrix},\begin{pmatrix} u_1 \\ \Theta (\lambda ) u_1 \end{pmatrix}\right\} \in \Gamma , \quad u_1 \in {{\mathfrak {L}}}_1,\,\, \lambda \in {{\mathcal {D}}}, \end{aligned}$$

  (3.50)

  and for all $\lambda \in {{\mathcal {D}}}_e$ the following relation holds

  $$\begin{aligned} \left\{ \begin{pmatrix} \gamma _2(\lambda )u_2 \\ \lambda \gamma _2(\lambda )u_2 \end{pmatrix},\begin{pmatrix} \Theta ^\#(\lambda )u_2 \\ u_2 \end{pmatrix}\right\} \in \Gamma , \quad u_2 \in {{\mathfrak {L}}}_2,\,\, \lambda \in {{\mathcal {D}}}_e, \end{aligned}$$

  (3.51)

  where

  $$\begin{aligned} \Theta ^\#(\lambda ):=\Theta (1/\overline{\lambda })^* \quad (\lambda \in {{\mathcal {D}}}_e). \end{aligned}$$

• (vi) Moreover, the following equalities are satisfied

  $$\begin{aligned} \gamma _1(\lambda )= & {} \gamma _1(\mu )+(\lambda -\mu )(V_1-\lambda I)^{-1}\gamma _1(\mu ),\quad \lambda ,\mu \in {{\mathcal {D}}}; \end{aligned}$$

  (3.52)
  $$\begin{aligned} \gamma _2(\lambda )= & {} \gamma _2(\mu )+(\lambda -\mu )(V_2-\lambda I )^{-1}\gamma _2(\mu ),\quad \lambda ,\mu \in {{\mathcal {D}}}_e. \end{aligned}$$

  (3.53)

Proof

• (i) By Proposition 3.7 there is $\alpha \in {{\mathbb {D}}}$, such that

  $$\begin{aligned} {\mathrm{ker\,}\,}(I-\alpha V_2)=\{0\}. \end{aligned}$$

  (3.54)

  Then the linear relation $V^{(\alpha )}=M^{(\alpha )}V$ is isometric and single-valued. By Lemma 3.14 the pair $({{\mathfrak {L}}},\Gamma ^{(\alpha )})$ with $\Gamma ^{(\alpha )}=\Gamma \circ M^{(-\alpha )}$ is a unitary boundary pair for the isometric operator $V^{(\alpha )}$.

  The linear relations $V_1^{(\alpha )}:={\mathrm{ker\,}\,}\Gamma _1^{(\alpha )}$, $V_2^{(\alpha )}:={\mathrm{ker\,}\,}\Gamma _2^{(\alpha )}$ are related with the linear relations $V_1$ and $V_2$ by the equalities

  $$\begin{aligned} V_1^{(\alpha )}= & {} M^{(\alpha )}(V_1)=\left\{ \begin{pmatrix} f-{\alpha }f' \\ -\overline{\alpha } f + f' \end{pmatrix}: \{f,f'\}\in V_1 \right\} , \end{aligned}$$

  (3.55)
  $$\begin{aligned} V_2^{(\alpha )}= & {} M^{(\alpha )}(V_2)=\left\{ \begin{pmatrix} f-{\alpha }f' \\ -\overline{\alpha } f + f' \end{pmatrix}: \{f,f'\}\in V_2 \right\} . \end{aligned}$$

  (3.56)

  By the choice of $\alpha$ we have $\mathrm{mul\,}V_2^{(\alpha )}=\{0\}$. Let ${{\mathcal {U}}}^{(\alpha )}={{\mathcal {J}}}\Gamma ^{(\alpha )}$. By Proposition 3.4$\mathrm{mul\,}{{\mathcal {U}}}^{(\alpha )}=\{0\}$, which by Lemma 2.5 (v) implies that ${{\mathcal {U}}}^{(\alpha )}$ is (the graph of) a unitary colligation $U^{(\alpha )}$. Now an application of Theorem 3.10 shows that $V_2^{(\alpha )}$ and $(V_1^{(\alpha )})^{-1}$ are graphs of closed bounded operators defined everywhere on ${{\mathfrak {H}}}$ and $V_1^{(\alpha )}=(V_2^{(\alpha )})^{-[*]}$; see (3.22). In view of (3.55) and (3.56) the linear relations $V_1$ and $V_2$ are closed and (3.38) implies that $V_1=V_2^{-[*]}$; cf. Lemma 3.13. Moreover, it follows from Lemma 3.13 that with $j=1,2$,

  $$\begin{aligned} \lambda \in \rho (V_j)\Leftrightarrow \mu ^{(\alpha )}(\lambda )\in \rho (V_j^{(\alpha )}). \end{aligned}$$

  (3.57)

  Hence the set ${{\mathcal {D}}}:=\rho (V_1)\cap {{\mathbb {D}}}$ is nonempty and $\mathrm{card\,}({{\mathbb {D}}}{\setminus }{{\mathcal {D}}})\le \kappa$ since the same properties hold for the set ${{\mathcal {D}}}^{(\alpha )}:=\rho (V_1^{(\alpha )})\cap {{\mathbb {D}}}$; see Theorem 3.10 (v).

  Similarly, by Theorem 3.10 (v) the set ${{\mathcal {D}}}_e^{(\alpha )}:=\rho (V_2^{(\alpha )})\cap {{\mathbb {D}}}_e$ is nonempty, $\mathrm{card\,}({{\mathbb {D}}}_e{\setminus }{{\mathcal {D}}}_e^{(\alpha )})\le \kappa$ and by (3.57) this implies the corresponding statement for ${{\mathbb {D}}}_e{\setminus }{{\mathcal {D}}}_e$.

• (ii) and (iii) By Theorem 3.11 the $\gamma$-field $\gamma _1^{(\alpha )}(\lambda )$ is holomorphic on ${{\mathcal {D}}}^{(\alpha )}$ with values in ${\mathbf {B}}({{\mathfrak {L}}}_1,{{\mathfrak {H}}})$ and the $\gamma$-field $\gamma _2^{(\alpha )}(\lambda )$ is holomorphic on ${{\mathcal {D}}}_e^{(\alpha )}$ with values in ${\mathbf {B}}({{\mathfrak {L}}}_2,{{\mathfrak {H}}})$. The desired statement for the $\gamma$-fields $\gamma _1(\lambda )$ and $\gamma _2(\lambda )$ follows from (3.42) and (3.43).

  The existence of the limits in (3.49) is obtained from (3.42) and Remark 3.12.

• (iv) This statement is implied by Theorem 3.11 (iii) and  (3.41).

• (v) The relation (3.50) is implied by (3.14), (3.15) and items (ii), (iv).

  To prove (3.51), let us choose for $\lambda \in {{\mathcal {D}}}_e$, $u_2\in {{\mathfrak {L}}}_2$, and a unique vector $v\in {{\mathfrak {L}}}_1$, such that

  $$\begin{aligned} \left\{ \begin{pmatrix} \gamma _2(\lambda )u_2 \\ \lambda \gamma _2(\lambda )u_2 \end{pmatrix},\begin{pmatrix} v \\ u_2 \end{pmatrix}\right\} \in \Gamma . \end{aligned}$$

  (3.58)

  By applying (3.2) or (3.9) to the elements (3.58) with $\lambda \in {{\mathcal {D}}}_e$ and (3.50) with $\lambda$ replaced by ${\bar{\lambda }}^{-1}\in {{\mathcal {D}}}$ it is seen that for all $u_1\in {{\mathfrak {L}}}_1$

  $$\begin{aligned} \begin{aligned} 0&=[\gamma _2(\lambda )u_2,\gamma _1({\bar{\lambda }}^{-1})u_1]_{{{\mathfrak {H}}}} - [\lambda \gamma _2(\lambda )u_2,{\bar{\lambda }}^{-1}\gamma _1({\bar{\lambda }}^{-1})u_1]_{{{\mathfrak {H}}}}\\&=(v,u_1)_{{{\mathfrak {L}}}_1} - (u_2,\Theta ({\bar{\lambda }}^{-1})u_1)_{{{\mathfrak {L}}}_2} \end{aligned} \end{aligned}$$

  and hence $v =\Theta ({\bar{\lambda }}^{-1})^*u_2=\Theta ^\#(\lambda )u_2$.

• (vi) To prove the identity (3.52) consider the vector $f_\mu =\gamma _1(\mu )u_1\in {{\mathfrak {N}}}_\mu (V_*)$, where $u_1\in {{\mathfrak {L}}}_1$, $\mu \in {{\mathcal {D}}}$. Then there exists $u_2\in {{\mathfrak {L}}}_2$ such that

  $$\begin{aligned} \left\{ \begin{pmatrix} f_\mu \\ \mu f_\mu \end{pmatrix},\begin{pmatrix} u_1 \\ u_2 \end{pmatrix}\right\} \in \Gamma ,\quad \mu \in {{\mathcal {D}}}. \end{aligned}$$

  (3.59)

  For $\lambda \in {{\mathcal {D}}}$ consider the vector

  $$\begin{aligned} {{\widehat{f}} }_\lambda =\begin{pmatrix} f_\mu \\ \mu f_\mu \end{pmatrix}+{{\widehat{g}} }, \quad where \quad {{\widehat{g}} }=(\lambda -\mu )\begin{pmatrix} (V_1-\lambda I)^{-1}f_\mu \\ I + \lambda (V_1-\lambda I)^{-1}f_\mu \end{pmatrix} \in V_1\subset V_*. \end{aligned}$$

  Direct calculations show that ${{\widehat{f}} }_\lambda \in {{\widehat{{\mathfrak {N}}}} }_\lambda (V_*)$. Since ${{\widehat{g}} }=\begin{pmatrix} g \\ g' \end{pmatrix}\in V_1$ there exists $v_2\in {{\mathfrak {L}}}_2$ such that

  $$\begin{aligned} \left\{ \begin{pmatrix} g \\ g' \end{pmatrix},\begin{pmatrix} 0 \\ v_2 \end{pmatrix}\right\} \in \Gamma . \end{aligned}$$

  (3.60)

  It follows from (3.59) and (3.60) that

  $$\begin{aligned} \left\{ \begin{pmatrix} f_\lambda \\ \lambda f_\lambda \end{pmatrix},\begin{pmatrix} u_1 \\ u_2+v_2 \end{pmatrix}\right\} = \left\{ \begin{pmatrix} f_\mu +g \\ \mu f_\mu +g' \end{pmatrix},\begin{pmatrix} u_1 \\ u_2+v_2 \end{pmatrix}\right\} \in \Gamma \end{aligned}$$

  and hence $\gamma _1(\lambda )u_1=f_\lambda$. This proves (3.52).

The equality (3.53) is proved similarly. $\square$

There is an analog for the notion of transposed boundary triple (see [19]) for boundary pairs of isometric operators. In the present case this notion contains the second boundary triple associated with a dual pair $\{V,V^{-1}\}$ as defined in [41] in the case of ordinary boundary triples for Hilbert space isometries. For this purpose the notion of transposed boundary pair $({{\mathfrak {L}}}_2\oplus {{\mathfrak {L}}}_1,\Gamma ^\top )$ is introduced for boundary pairs of isometric operators and its basic properties are established in the next proposition.

Proposition 3.16

Let $({{\mathfrak {L}}},\Gamma )$ be a unitary boundary pair for an isometric operator V in a Pontryagin space ${{\mathfrak {H}}}$, let $V_1={\mathrm{ker\,}\,}\Gamma _1$, $V_2={\mathrm{ker\,}\,}\Gamma _2$, let $\gamma _1(\lambda )$ and $\gamma _2(\lambda )$ be the $\gamma$-fields of $({{\mathfrak {L}}},\Gamma )$, let $\Theta (\lambda )$ be the Weyl function of $({{\mathfrak {L}}},\Gamma )$, and define

$$\begin{aligned} \Gamma ^\top =\left\{ \left\{ \left( {\begin{array}{c}f'\\ f\end{array}}\right) ,\left( {\begin{array}{c}u_2\\ u_1\end{array}}\right) \right\} :\, \left\{ \left( {\begin{array}{c}f\\ f'\end{array}}\right) ,\left( {\begin{array}{c}u_1\\ u_2\end{array}}\right) \right\} \in \Gamma \right\} . \end{aligned}$$

(3.61)

Then:

• (i) $({{\mathfrak {L}}}_2\oplus {{\mathfrak {L}}}_1,\Gamma ^\top )$ is a unitary boundary pair for $V^{-1}$.

• (ii) $V_1^\top :={\mathrm{ker\,}\,}\Gamma _1^\top =V_2^{-1}$ and $V_2^\top :={\mathrm{ker\,}\,}\Gamma _2^\top =V_1^{-1}$.

• (iii) The Weyl function $\Theta ^\top (\lambda )$ and the $\gamma$-fields $\gamma _1^\top (\lambda )$ and $\gamma _2^\top (\lambda )$, corresponding to the pair $({{\mathfrak {L}}}_2\oplus {{\mathfrak {L}}}_1,\Gamma ^\top )$ are connected with $\Theta (\lambda )$, $\gamma _1(\lambda )$ and $\gamma _2(\lambda )$ by

  $$\begin{aligned} \Theta ^\top (\lambda )= & {} \Theta ({\bar{\lambda }})^*,\quad \lambda \in \overline{{{\mathcal {D}}}}:=\{\overline{\mu }:\mu \in {{\mathcal {D}}}\}, \end{aligned}$$

  (3.62)
  $$\begin{aligned} \gamma _1^\top (\lambda )= & {} \frac{1}{\lambda }\gamma _2\left( \frac{1}{\lambda }\right) ,\quad \lambda \in \overline{{{\mathcal {D}}}};\qquad \gamma _2^\top (\lambda )=\frac{1}{\lambda }\gamma _1\left( \frac{1}{\lambda }\right) ,\quad \lambda \in \overline{{{\mathcal {D}}}}_e. \end{aligned}$$

  (3.63)

Proof

• (i) This statement is implied by the equality $V={\mathrm{ker\,}\,}\Gamma$ and the following identity

  $$\begin{aligned}{}[f',g']_{{{{\mathfrak {H}}}}}-[f,g]_{{{{\mathfrak {H}}}}}=(u_2,v_2)_{{{{\mathfrak {L}}}}_2}-(u_1,v_1)_{{{{\mathfrak {L}}}}_1}, \end{aligned}$$

  (3.64)

  which is valid for all $\{{{\widehat{f}} }, {{\widehat{u}} }\},\{{{\widehat{g}} }, {{\widehat{v}} }\}\in \Gamma$; see Definition 3.1.

• (ii) The statement is clear from the definition of $({{\mathfrak {L}}}_2\oplus {{\mathfrak {L}}}_1,\Gamma ^\top )$.

• (iii) If $\lambda \in \overline{{{\mathcal {D}}}}$ then $1/\lambda \in {{{\mathcal {D}}}_e}$ and one obtains from (3.51)

  $$\begin{aligned} \left\{ \begin{pmatrix} \gamma _2(1/\lambda )u_2 \\ 1/\lambda \gamma _2(1/\lambda )u_2 \end{pmatrix},\begin{pmatrix} \Theta ^\#(1/\lambda )u_2 \\ u_2 \end{pmatrix}\right\} \in \Gamma ,\quad u_2\in {{\mathfrak {L}}}_2,\quad \lambda \in \overline{{{\mathcal {D}}}}. \end{aligned}$$

  In view of (3.61) this implies

  $$\begin{aligned} \left\{ \begin{pmatrix} 1/\lambda \gamma _2(1/\lambda )u_2 \\ \gamma _2(1/\lambda )u_2 \end{pmatrix},\begin{pmatrix} u_2\\ \Theta ^\#(1/\lambda )u_2 \end{pmatrix}\right\} \in \Gamma ^\top ,\quad u_2\in {{\mathfrak {L}}}_2,\quad \lambda \in \overline{{{\mathcal {D}}}}, \end{aligned}$$

  and hence

  $$\begin{aligned} \gamma _1^\top (\lambda )=\frac{1}{\lambda }\gamma _2\left( \frac{1}{\lambda }\right) \quad \text{ and } \quad \Theta ^\top (\lambda )=\Theta ^\#(1/\lambda )=\Theta ({\bar{\lambda }})^*,\quad \lambda \in \overline{{{\mathcal {D}}}}. \end{aligned}$$

  The second equality in (3.63) is proved similarly. $\square$

3.5 Realization Theorem

The converse statement to Theorem 3.15 contains the main realization result for the generalized Schur class ${{\mathcal {S}}}_\kappa ({{\mathfrak {L}}}_1,{{\mathfrak {L}}}_2)$: every function $\Theta (\lambda )$ from the class ${{\mathcal {S}}}_\kappa ({{\mathfrak {L}}}_1,{{\mathfrak {L}}}_2)$ can be realized as the Weyl function of a boundary pair for some isometric operator V in a Pontryagin space.

Theorem 3.17

Let $s(\cdot )\in {{\mathcal {S}}}_\kappa ({{{\mathfrak {L}}}}_1,{{{\mathfrak {L}}}}_2)$ with the domain of holomorphy ${{\mathfrak {h}}}_s(\subset {{\mathbb {D}}})$. Then there exists a simple isometric operator V in a Pontryagin space ${{{\mathfrak {H}}}}$ and a unitary boundary pair $({{{\mathfrak {L}}}}_1\times {{{\mathfrak {L}}}}_2, \Gamma )$ such that the corresponding Weyl function $\Theta (\lambda )$ coincides with $s(\lambda )$ on ${{\mathfrak {h}}}_s$.

Proof

• (1) First assume that $s(\cdot )$ is holomorphic at 0. Then by [1, Theorem 2.3.1] there exists a closely connected unitary colligation $U=\begin{pmatrix} T&{} F\\ G&{} H \end{pmatrix}$ such that its characteristic function coincides with $s(\lambda )$ for all $\lambda \in {{\mathfrak {h}}}_s$.

  Let V be defined by the formula $V=T^{[*]}\upharpoonright {\mathrm{ker\,}\,}F^{[*]}$ and let the linear relation $\Gamma$ be defined by (3.18). Then by Theorem  3.10 the pair $({{{\mathfrak {L}}}},\Gamma )$ is a unitary boundary pair for $V_*$ and the corresponding Weyl function $\Theta (\lambda )$ coincides with $s(\lambda )$ on ${{\mathfrak {h}}}_s$.

• (2) Assume that $s(\lambda )$ is holomorphic at $\overline{\alpha }\in {{\mathfrak {h}}}_s$. Consider a new operator function

  $$\begin{aligned} s^{(\alpha )}(\zeta )=s\left(\frac{\zeta +\overline{\alpha }}{1+ \alpha \zeta }\right). \end{aligned}$$

  (3.65)

  Since $s^{(\alpha )}(\cdot )\in {{\mathcal {S}}}_\kappa ({{{\mathfrak {L}}}}_1,{{{\mathfrak {L}}}}_2)$ and also is holomorphic at 0 there exist a Pontryagin space ${{{\mathfrak {H}}}}$, a simple isometric operator $V^{(\alpha )}$ in ${{{\mathfrak {H}}}}$, and a unitary boundary pair $({{{\mathfrak {L}}}}, \Gamma ^{(\alpha )})$ such that the corresponding Weyl function $\Theta ^{(\alpha )}(\zeta )$ coincides with $s^{(\alpha )}(\zeta )$.

  By Lemma 3.14 the pair $({{\mathfrak {L}}},\Gamma )$ with $\Gamma :=\Gamma ^{(\alpha )} \circ M^{(\alpha )}$ is a unitary boundary pair for the simple isometric operator

  $$\begin{aligned} V={\mathrm{ker\,}\,}\Gamma =M^{(-\alpha )}V^{(\alpha )}= \left\{ \begin{pmatrix} f+{\alpha }f' \\ \overline{\alpha } f + f' \end{pmatrix}: \{f,f'\}\in V^{(\alpha )} \right\} . \end{aligned}$$

  The domains $V_*:=\mathrm{dom\,}\Gamma$ and $V_*^{(\alpha )}:=\mathrm{dom\,}\Gamma ^{(\alpha )}$ of $\Gamma$ and $\Gamma ^{(\alpha )}$ are connected by

  $$\begin{aligned} V_*:=\mathrm{dom\,}\Gamma =M^{(-\alpha )}V_*^{(\alpha )}= \left\{ \begin{pmatrix} f+{\alpha }f' \\ \overline{\alpha } f + f' \end{pmatrix}: \{f,f'\}\in V_*^{(\alpha )} \right\} , \end{aligned}$$

  The defect subspace ${{\mathfrak {N}}}_\lambda (V_*):={\mathrm{ker\,}\,}(V_*-\lambda I)$ and ${{\mathfrak {N}}}_\omega (V_*^{(\alpha )}):={\mathrm{ker\,}\,}(V_*^{(\alpha )}-\omega I)$ are connected by (3.45) and the equivalence (3.46) holds. It follows from (3.46) and Definition 3.9 that the Weyl functions of the unitary boundary pairs $({{{\mathfrak {L}}}},\Gamma )$ and $({{{\mathfrak {L}}}},\Gamma ^{(\alpha )})$ are connected by

  $$\begin{aligned} \Theta (\lambda ) =\Theta ^{(\alpha )} \left( \frac{\lambda -\overline{\alpha }}{1-{\alpha }\lambda }\right) ,\quad \lambda \in {{\mathfrak {h}}}_s. \end{aligned}$$

  (3.66)

  Setting $\zeta =\frac{\lambda -\overline{\alpha }}{1-{\alpha }\lambda }$ one obtains $\lambda ={\frac{\zeta +\overline{\alpha }}{1+ \alpha \zeta }}$ and hence by (3.66) and (3.65)

  $$\begin{aligned} \Theta (\lambda )=\Theta ^{(\alpha )}(\zeta )=s^{(\alpha )}(\zeta )=s\left(\frac{\zeta +\overline{\alpha }}{1+ \alpha \zeta }\right)=s(\lambda ),\quad \lambda \in {{\mathfrak {h}}}_s. \end{aligned}$$

  This completes the proof. $\square$

Remark 3.18

Notice that a simple isometric operator V in Theorem 3.17 and a unitary boundary pair $({{{\mathfrak {L}}}}_1\times {{{\mathfrak {L}}}}_2, \Gamma )$ are determined by the Weyl function $s(\cdot )$ uniquely, up to a unitary equivalence. The latter means that if there is another simple isometric operator $V'$ in a Pontryagin space ${{{\mathfrak {H}}}}'$ and a unitary boundary pair $({{{\mathfrak {L}}}}_1\times {{{\mathfrak {L}}}}_2, \Gamma ')$, such that its Weyl function coincides with $s(\cdot )$, then there exists a unitary operator W from ${{{\mathfrak {H}}}}'$ to ${{{\mathfrak {H}}}}'$, such that

$$\begin{aligned} V'=WVW^{-1},\quad \Gamma '=\Gamma W^{-1}. \end{aligned}$$

Next Theorem 3.15 is specialized to the case where ${{\mathfrak {H}}}$ is a Hilbert space and $({{\mathfrak {L}}},\Gamma )$ is a unitary boundary pair for an isometric operator V in ${{\mathfrak {H}}}$. When combined with Theorem 3.17 we get a general realization result for operator valued Schur functions $s(\cdot )\in {{\mathcal {S}}}({{{\mathfrak {L}}}}_1,{{{\mathfrak {L}}}}_2)$ as a Weyl function of a unitary boundary pairs for an isometric operator in a Hilbert space.

Theorem 3.19

Let ${{\mathfrak {H}}}$ and ${{\mathfrak {L}}}={{\mathfrak {L}}}_1\times {{\mathfrak {L}}}_2$ be Hilbert spaces, let V be an isometric operator in ${{\mathfrak {H}}}$, let $({{\mathfrak {L}}},\Gamma )$ be a unitary boundary pair for V and let $V_1$, $V_2$ be defined by (3.5). Then:

• (i) $V_1^{-1}$ and $V_2$ are contractive operators in ${\mathbf {B}}({{\mathfrak {H}}})$ and they are connected by

  $$\begin{aligned} V_2=V_1^{-*}. \end{aligned}$$

  (3.67)

• (ii) The $\gamma$-field $\gamma _1(\lambda )$ is holomorphic on ${{\mathbb {D}}}$ with values in ${\mathbf {B}}({{\mathfrak {L}}}_1,{{\mathfrak {H}}})$ and satisfies the identity (3.52) for all $\lambda ,\mu \in {{\mathbb {D}}}$.

• (iii) The $\gamma$-field $\gamma _2(\lambda )$ is holomorphic on ${{\mathbb {D}}}_e$ with values in ${\mathbf {B}}({{\mathfrak {L}}}_2,{{\mathfrak {H}}})$ and satisfies the identity (3.53) for all $\lambda ,\mu \in {{\mathbb {D}}}_e$.

• (iv) The Weyl function $\Theta (\lambda )$ of V corresponding to the boundary pair $({{\mathfrak {L}}}, \Gamma )$ belongs to the Schur class ${{\mathcal {S}}}({{\mathfrak {L}}}_1,{{\mathfrak {L}}}_2)$.

Conversely, for every function $s(\cdot )$ from the Schur class ${{\mathcal {S}}}({{{\mathfrak {L}}}}_1,{{{\mathfrak {L}}}}_2)$ there exists a simple isometric operator V in a Hilbert space ${{{\mathfrak {H}}}}$ and a unitary boundary pair $({{\mathfrak {L}}}, \Gamma )$ for V such that the corresponding Weyl function $\Theta (\cdot )$ coincides with $s(\cdot )$.

Proof

(i) According to Corollary 3.5$V_1^{-1}$ and $V_2$ are contractive operators. From item (i) in Theorem 3.15 one concludes that

$$\begin{aligned} {{\mathbb {D}}}\subseteq \rho (V_2^{-1}) \quad and \quad {{\mathbb {D}}}_e\subseteq \rho (V_1^{-1}). \end{aligned}$$

In particular, $0\in \rho (V_2^{-1})$ so that $V_2\in {\mathbf {B}}({{\mathfrak {H}}})$. On the other hand, by Proposition 3.4 one has the inclusion $V_2\subseteq V_1^{-*}$. Since ${{\mathbb {D}}}_e\subseteq \rho (V_1^{-*})$, the equality (3.67) must prevail and, hence, $V_1^{-*}\in {\mathbf {B}}({{\mathfrak {H}}})$ $\Leftrightarrow$ $V_1^{-1}\in {\mathbf {B}}({{\mathfrak {H}}})$.

The assertions (ii)–(iv) are now obtained directly from Theorem 3.15.

The last statement follows from Theorem 3.17 by taking $\kappa =0$. $\square$

3.6 Classification of Unitary Boundary Pairs

We start by collecting some main properties of unitary boundary pairs in the next proposition.

Proposition 3.20

Let V be an isometric operator in the Pontryagin space ${{{\mathfrak {H}}}}$ and let $({{{\mathfrak {L}}}},\Gamma )$ be a unitary boundary pair for V. Then $\Gamma$ and its components $\Gamma _1$ and $\Gamma _2$ defined by (3.3), (3.4) admit the following properties:

• (i) $\Gamma _1$ and $\Gamma _2$ are closed linear relations with $\mathrm{ran\,}\Gamma _1={{\mathfrak {L}}}_1$, $\mathrm{ran\,}\Gamma _2={{\mathfrak {L}}}_2$ and, moreover, $\mathrm{mul\,}\Gamma _1=\{0\}$ $\Leftrightarrow$ $\mathrm{mul\,}\Gamma _2=\{0\}$ $\Leftrightarrow$ $\mathrm{mul\,}\Gamma =\{0\}$;

• (ii) $V_1={\mathrm{ker\,}\,}\Gamma _1$ and $V_2={\mathrm{ker\,}\,}\Gamma _2$ have nonempty resolvent sets and $V_1=V_2^{-[*]}$;

• (iii) $V_*=\mathrm{dom\,}\Gamma$ admits the decompositions

  $$\begin{aligned} V_*= & {} V_1 {{\,\mathrm{\, \widehat{+} \,}\,}}{{\widehat{{\mathfrak {N}}}} }_\lambda (V_*), \quad \lambda \in {{\mathcal {D}}}=\rho (V_1)\cap {{\mathbb {D}}},\\ V_*= & {} V_2 {{\,\mathrm{\, \widehat{+} \,}\,}}{{\widehat{{\mathfrak {N}}}} }_\lambda (V_*), \quad \lambda \in {{\mathcal {D}}}_e=\rho (V_2)\cap {{\mathbb {D}}}_e . \end{aligned}$$

Proof

• (i) These properties were proven in Proposition 3.4.

• (ii) The fact that $\rho (V_1)$ and $\rho (V_2)$ are nonempty and the equality $V_2=V_1^{-*}$ were proven in Theorem 3.15.

• (iii) This is a direct consequence of (ii); see e.g. [29, Lemma 4.1]. $\square$

Remark 3.21

Proposition 3.20 shows that in a Pontryagin space ${{{\mathfrak {H}}}}$ every unitary boundary pair $({{{\mathfrak {L}}}},\Gamma )$ (as well as its transposed boundary pair) of an isometric operator V can be seen as an analog of so-called (B-)generalized boundary triple, since the component map**s $\Gamma _1$ and $\Gamma _2$ are surjective and the corresponding kernels $V_1={\mathrm{ker\,}\,}\Gamma _1$ and $V_2={\mathrm{ker\,}\,}\Gamma _2$ are closed extensions of V with nonempty resolvent sets; see [19, 26] and [22,23,24] for some further developments.

Proposition 3.22

The following relations hold:

$$\begin{aligned} {{\textsf {S}}}^\Theta _\mu (\lambda ):= & {} \frac{I-\Theta (\mu )^* \Theta (\lambda )}{1-\lambda \overline{\mu }}=\gamma _1(\mu )^{[*]}\gamma _1(\lambda ),\quad \lambda ,\mu \in {{\mathcal {D}}}; \end{aligned}$$

(3.68)

$$\begin{aligned} {{\textsf {S}}}^\Theta _\mu (\lambda ):= & {} \frac{I-\Theta ^\#(\mu )^* \Theta ^\#(\lambda )}{1-\lambda \overline{\mu }}=-\gamma _2(\mu )^{[*]}\gamma _2(\lambda ),\quad \lambda ,\mu \in {{\mathcal {D}}}_e; \end{aligned}$$

(3.69)

$$\begin{aligned} {{\textsf {S}}}^\Theta _\mu (\lambda ):= & {} \frac{\Theta (\lambda )-\Theta ^\#(\mu )^* }{1-\lambda \overline{\mu }}=-\gamma _2(\mu )^{[*]}\gamma _1(\lambda ),\quad \lambda \in {{\mathcal {D}}},\text { }\mu \in {{\mathcal {D}}}_e; \end{aligned}$$

(3.70)

$$\begin{aligned} {{\textsf {S}}}^\Theta _\mu (\lambda ):= & {} \frac{\Theta ^\#(\lambda )-\Theta (\mu )^*}{1-\lambda \overline{\mu }}=\gamma _1(\mu )^{[*]}\gamma _2(\lambda ),\quad \lambda \in {{\mathcal {D}}}_e, \text { }\mu \in {{\mathcal {D}}}. \end{aligned}$$

(3.71)

Proof

Let $\lambda ,\mu \in {{\mathcal {D}}}$ and $u_1,v_1\in {{\mathfrak {L}}}_1$. Then by (3.50)

$$\begin{aligned} \left\{ \begin{pmatrix} \gamma _1(\lambda )u_1 \\ \lambda \gamma _1(\lambda )u_1 \end{pmatrix}, \begin{pmatrix} u_1 \\ \Theta (\lambda )u_1 \end{pmatrix}\right\} \in \Gamma ,\quad \left\{ \begin{pmatrix} \gamma _1(\mu )v_1 \\ \mu \gamma _1(\mu )v_1 \end{pmatrix}, \begin{pmatrix} v_1 \\ \Theta (\mu )v_1 \end{pmatrix}\right\} \in \Gamma \end{aligned}$$

and the identity (3.9) applied to these vectors yields

$$\begin{aligned} (1-\lambda \overline{\mu })[\gamma _1(\lambda )u_1,\gamma _1(\mu )v_1]_{{\mathfrak {H}}}=[u_1,v_1]_{{{\mathfrak {L}}}_1} - [\Theta (\lambda )u_1,\Theta (\mu )v_1]_{{{\mathfrak {L}}}_2}. \end{aligned}$$

This proves the equality (3.68).

Let now $\lambda \in {{\mathcal {D}}}$, $\mu \in {{\mathcal {D}}}_e$ and let $u_1\in {{\mathfrak {L}}}_1$ and $v_2\in {{\mathfrak {L}}}_2$. Then by (3.50)

$$\begin{aligned} \left\{ \begin{pmatrix} \gamma _1(\lambda )u_1 \\ \lambda \gamma _1(\lambda )u_1 \end{pmatrix}, \begin{pmatrix} u_1 \\ \Theta (\lambda )u_1 \end{pmatrix}\right\} \in \Gamma ,\quad \left\{ \begin{pmatrix} \gamma _2(\mu )v_2 \\ \mu \gamma _2(\mu )v_2 \end{pmatrix}, \begin{pmatrix} \Theta ^\#(\mu )v_2 \\ v_2 \end{pmatrix}\right\} \in \Gamma . \end{aligned}$$

By applying (3.9) to these vectors one arrives at

$$\begin{aligned} (1-\lambda \overline{\mu })[\gamma _1(\lambda )u_1,\gamma _2(\mu )v_2]_{{\mathfrak {H}}}=[u_1,\Theta ^\#(\mu )v_2]_{{{\mathfrak {L}}}_1} - [\Theta (\lambda )u_1,v_2]_{{{\mathfrak {L}}}_2}. \end{aligned}$$

This yields (3.70). The proof of (3.69) and (3.71) is analogous. $\square$

Proposition 3.23

Let $({{{\mathfrak {L}}}},\Gamma )$ be a unitary boundary pair for an isometric operator V in the Pontryagin space ${{{\mathfrak {H}}}}$. Then

$$\begin{aligned} \mathrm{mul\,}\Gamma _1={\mathrm{ker\,}\,}\gamma _1(\lambda ) \; (\lambda \in {{\mathcal {D}}}) \quad \text {and}\quad \mathrm{mul\,}\Gamma _2={\mathrm{ker\,}\,}\gamma _2(\lambda ) \; (\lambda \in {{\mathcal {D}}}_e). \end{aligned}$$

Moreover, the following statements are equivalent:

• (i) $\Gamma$ is single valued (i.e. $\mathrm{mul\,}\Gamma =\{0\}$);

• (ii) $\mathrm{ran\,}\Gamma$ is dense in ${{\mathfrak {L}}}_1\times {{\mathfrak {L}}}_2$;

• (iii) ${\mathrm{ker\,}\,}\gamma _1(\lambda )=\{0\}$ for some (equivalently for all) $\lambda \in {{\mathcal {D}}}$;

• (iv) ${\mathrm{ker\,}\,}\gamma _2(\lambda )=\{0\}$ for some (equivalently for all) $\lambda \in {{\mathcal {D}}}_e$.

If, in addition, the operator V is simple then the conditions (i)–(iv) are equivalent to

• (v) $\bigcap \limits _{\mu \in {{\mathcal {D}}}\cup {{\mathcal {D}}}_e}{\mathrm{ker\,}\,}{{\textsf {S}}}^\Theta _\mu (\lambda )=\{0\}$ for some (equivalently for all) $\lambda \in {{\mathcal {D}}}\cup {{\mathcal {D}}}_e$.

Proof

(i) $\Leftrightarrow$ (ii) This is item (ii) in Proposition 3.2.

(i) $\Leftrightarrow$ (iii), (iv) The formula (3.11) and Definition 3.8 of the $\gamma$-fields shows that ${\mathrm{ker\,}\,}\gamma _1(\lambda )=\mathrm{mul\,}\Gamma _1$ for $\lambda \in {{\mathcal {D}}}$ and ${\mathrm{ker\,}\,}\gamma _2(\lambda )=\mathrm{mul\,}\Gamma _2$ for $\lambda \in {{\mathcal {D}}}_e$. Now the statement follows from item (i) in Proposition 3.20.

To prove the statement in (v) first observe that with $\lambda \in {{\mathcal {D}}}$ the inclusions

$$\begin{aligned} {\mathrm{ker\,}\,}\gamma _1(\lambda )\subseteq {\mathrm{ker\,}\,}{{\textsf {S}}}^\Theta _\mu (\lambda )\; for all \; \mu \in {{\mathcal {D}}}\cup {{\mathcal {D}}}_e \end{aligned}$$

are clear from (3.68) and (3.70) in Proposition 3.22. Thus

$$\begin{aligned} {\mathrm{ker\,}\,}\gamma _1(\lambda )\subseteq \bigcap \limits _{\mu \in {{\mathcal {D}}}\cup {{\mathcal {D}}}_e}{\mathrm{ker\,}\,}{{\textsf {S}}}^\Theta _\mu (\lambda ), \quad \lambda \in {{\mathcal {D}}}. \end{aligned}$$

(3.72)

As to the reverse inclusion apply Proposition 3.22 again to see that for all $\lambda ,\mu \in {{\mathcal {D}}}$ and $u_1,v_1\in {{\mathfrak {L}}}_1$,

$$\begin{aligned} \left( {{\textsf {S}}}^\Theta _\mu (\lambda )u_1,v_1\right) _{{{\mathfrak {L}}}_1}=[\gamma _1(\lambda )u_1,\gamma _1(\mu )v_1]_{{\mathfrak {H}}}\end{aligned}$$

(3.73)

and for all $\lambda \in {{\mathcal {D}}}$, $\nu \in {{\mathcal {D}}}_e$ and $u_1\in {{\mathfrak {L}}}_1$, $v_2\in {{\mathfrak {L}}}_2$,

$$\begin{aligned} \left( {{\textsf {S}}}^\Theta _\mu (\lambda )u_1,v_2\right) _{{{\mathfrak {L}}}_2}=[\gamma _1(\lambda )u_1,\gamma _2(\mu )v_2]_{{\mathfrak {H}}}. \end{aligned}$$

(3.74)

Now assume that the operator V is simple. Then

$$\begin{aligned} \overline{\mathrm{span\,}}\{\gamma _1(\mu ){{\mathfrak {L}}}_1,\, \gamma _2(\nu ){{\mathfrak {L}}}_2:\,\mu \in {{\mathcal {D}}},\,\nu \in {{\mathcal {D}}}_e\}={{\mathfrak {H}}}\end{aligned}$$

and by virtue of (3.73) and (3.74) the reverse inclusion in (3.72) follows. This proves the equivalence (iv) $\Leftrightarrow$ (v) for $\lambda \in {{\mathcal {D}}}$ when V is simple. Similarly one proves the equivalence (iii) $\Leftrightarrow$ (v) for $\lambda \in {{\mathcal {D}}}_e$. $\square$

Proposition 3.24

Let $({{{\mathfrak {L}}}},\Gamma )$ be a unitary boundary pair for an isometric operator V in the Pontryagin space ${{{\mathfrak {H}}}}$. Then the following statements are equivalent:

• (i) $\mathrm{ran\,}\Gamma ={{\mathfrak {L}}}_1\times {{\mathfrak {L}}}_2$;

• (ii) $\mathrm{dom\,}\Gamma =V^{-[*]}$ and $\mathrm{mul\,}\Gamma =\{0\}$;

• (iii) ${\mathrm{ker\,}\,}\gamma _1(\lambda )=\{0\}$ and $\mathrm{ran\,}\gamma _1(\lambda )$ is closed for some (equivalently for all) $\lambda \in {{\mathcal {D}}}$;

• (iv) ${\mathrm{ker\,}\,}\gamma _2(\lambda )=\{0\}$ and $\mathrm{ran\,}\gamma _2(\lambda )$ is closed for some (equivalently for all) $\lambda \in {{\mathcal {D}}}_e$.

If $0\in \rho ({{\textsf {S}}}^\Theta _\lambda (\lambda ))$ for some $\lambda \in {{\mathcal {D}}}\cup {{\mathcal {D}}}_e$, then the conditions (i)–(iv) hold.

If one of the condition (i)–(iv) is satisfied then $({{{\mathfrak {L}}}},\Gamma )$ is an ordinary boundary pair for V.

Proof

(i) $\Leftrightarrow$ (ii) This is item (iii) in Proposition 3.2.

(ii) $\Leftrightarrow$ (iii), (iv) Definition 3.8 of the $\gamma$-fields shows that $\mathrm{ran\,}\gamma _1(\lambda )={{\mathfrak {N}}}_\lambda (V_*)$ for $\lambda \in {{\mathcal {D}}}$ and $\mathrm{ran\,}\gamma _2(\lambda )={{\mathfrak {N}}}_\lambda (V_*)$ for $\lambda \in {{\mathcal {D}}}_e$. Now the decompositions of $V_*=\mathrm{dom\,}\Gamma$ in item (iii) of Proposition 3.20 imply that ${{\mathfrak {N}}}_\lambda (V_*)$ is closed for some $\lambda \in {{\mathcal {D}}}\cup {{\mathcal {D}}}_e$ if and only if $V_*=V^{-[*]}$. This combined with Proposition 3.23 gives the stated equivalences.

The last implication follows from Proposition 3.22. Indeed, if $0\in \rho ({{\textsf {S}}}^\Theta _\lambda (\lambda ))$ for some $\lambda \in {{\mathcal {D}}}$, then by Proposition 3.23${\mathrm{ker\,}\,}\gamma _1(\lambda )=\{0\}$ and by (3.68) $\mathrm{ran\,}\gamma _1(\lambda )$ is closed. Similarly, if $0\in \rho ({{\textsf {S}}}^\Theta _\lambda (\lambda ))$ for some $\lambda \in {{\mathcal {D}}}_e$, then again by Proposition 3.23${\mathrm{ker\,}\,}\gamma _2(\lambda )=\{0\}$ and by (3.69) $\mathrm{ran\,}\gamma _2(\lambda )$ is closed.

Finally, the fact that $({{{\mathfrak {L}}}},\Gamma )$ is an ordinary boundary pair for V is clear from the properties in (ii). $\square$

The next example shows that the condition (v) in Proposition 3.23 cannot be replaced by a single condition ${\mathrm{ker\,}\,}{{\textsf {S}}}^\Theta _\lambda (\lambda )=\{0\}$.

Example 3.25

Let ${{\mathfrak {H}}}={{\mathbb {C}}}^4$ with the skew-diagonal fundamental symmetry $J=(\delta _{j,5-k})_{j,k=1}^4$, where $\delta _{j,k}$ is the Kronecker delta and let us set $e_j=(\delta _{j,k})_{k=1}^4$, $(j=1,2,3,4)$. Let V be an isometry in ${{\mathfrak {H}}}$ which maps $e_1$ into $e_2$. Then the defect subspaces of V

$$\begin{aligned} {{\mathfrak {N}}}_\lambda =\left\{ (f_1,f_2, f_3,\lambda f_3)^\top :\, f_1,f_2,f_3\in {{\mathbb {C}}}\right\} \end{aligned}$$

(3.75)

are degenerate for all $\lambda \in {{\mathbb {C}}}$ and thus, the operator V is not standard.

The linear relation $V^{-[*]}$ consists of vectors

$$\begin{aligned} {{\widehat{f}} }=\begin{pmatrix} f \\ f' \end{pmatrix},\quad \text{ where } \quad f=\sum _{j=1}^{4}f_j e_j,\quad f'=\sum _{j=1}^{4}f'_j e_j,\quad f_3'=f_4. \end{aligned}$$

Therefore, the left part of the identity (3.2) for ${{\widehat{f}} }={{\widehat{g}} }$ takes the form

$$\begin{aligned} (f_1-f_2')\overline{f_4} +f_2\overline{f_3}+f_3\overline{f_2}+f_4(\overline{f_1}-\overline{f_2'})-f_1'\overline{f_4'}-f_4'\overline{f_1'} \end{aligned}$$

and can be rewritten in the diagonal form

$$\begin{aligned}&\frac{1}{2}\quad \left\{ |f_1'-\!f_4'|^2-\!|f_1'+f_4'|^2 + |f_1-\! f_2'+f_4|^2\quad -\! |f_1-f_2'-\!f_4|^2\right. \\&\quad \left. +|f_2+f_3|^2 -\! |f_2-\! f_3|^2\right\} . \end{aligned}$$

Hence a single-valued boundary triple $({{\mathfrak {L}}}_1\oplus {{\mathfrak {L}}}_2, \Gamma _1,\Gamma _2)$ can be chosen as follows

$$\begin{aligned} {{\mathfrak {L}}}_1={{\mathfrak {L}}}_2={{\mathbb {C}}}^3,\quad \Gamma _1 {{\widehat{f}} }=\frac{1}{\sqrt{2}}\begin{pmatrix} f'_1-f'_4\\ f_1-f'_2+f_4 \\ f_2+f_3 \end{pmatrix},\quad \Gamma _2 {{\widehat{f}} }=\frac{1}{\sqrt{2}}\begin{pmatrix} f'_1+f'_4\\ f_1-f'_2-f_4\\ f_2-f_3 \end{pmatrix}.\nonumber \\ \end{aligned}$$

(3.76)

Then for $|\lambda |<1$ one obtains from (3.75) and (3.76)

$$\begin{aligned} \Theta (\lambda )=\frac{1}{3\lambda ^2}\begin{pmatrix} \lambda ^2&{}2\lambda ^3&{}2\lambda ^4 \\ 2\lambda &{}\lambda ^2&{}-2\lambda ^3 \\ 2&{}-2\lambda &{}\lambda ^2 \end{pmatrix} \end{aligned}$$

and hence

$$\begin{aligned} {{\textsf {S}}}^\Theta _\omega (\lambda )=\frac{-2}{9\lambda ^2\overline{\omega }^2}\Omega ^* \begin{pmatrix} 2(1+2\lambda \overline{\omega })&{}-(2+\lambda \overline{\omega })&{}1-\lambda \overline{\omega }\\ -(2+\lambda \overline{\omega })&{}2(1-\lambda \overline{\omega })&{}-(1+2\lambda \overline{\omega }) \\ 1-\lambda \overline{\omega }&{}-(1+2\lambda \overline{\omega })&{}-2(2+\lambda \overline{\omega }) \end{pmatrix}\Lambda , \end{aligned}$$

where $\Lambda =\text{ diag } (1,\lambda ,\lambda ^2)$, $\Omega =\text{ diag } (1,\omega ,\omega ^2)$. Notice that in this example $\det {{\textsf {S}}}^\Theta _\omega (\lambda )\equiv 0$ for all $\lambda ,\omega \in {{\mathbb {D}}}\backslash \{0\}$, while $\bigcap \limits _{\omega \in {{\mathbb {D}}}\backslash \{0\}}{\mathrm{ker\,}\,}{{\textsf {S}}}^\Theta _{\omega }(\lambda )=\{0\}$. In fact, for every pair $\omega _1,\omega _2 \in {{\mathbb {D}}}\backslash \{0\}$, $\omega _1\ne \omega _2$, one gets ${\mathrm{ker\,}\,}{{\textsf {S}}}^\Theta _{\omega _1}(\lambda )\cap {\mathrm{ker\,}\,}{{\textsf {S}}}^\Theta _{\omega _2}(\lambda )=\{0\}$.

Remark 3.26

Let A be a closed symmetric operator in a Pontryagin space ${{\mathfrak {H}}}$ with equal defect numbers, and let $\pm i\notin \sigma _p(A)$. Then its Cayley transform $V=(A-iI)(A+iI)^{-1}$ is an isometric operator in ${{\mathfrak {H}}}$.

Let $({{\mathfrak {L}}}_1\oplus {{\mathfrak {L}}}_2,\Gamma )$ be a boundary pair for V with $V_*=\mathrm{dom\,}\Gamma$ such that ${{\mathfrak {L}}}_1={{\mathfrak {L}}}_2=:{{\mathcal {H}}}$. Define the Kreĭn spaces $({{\mathfrak {H}}}^2,[\![\cdot ,\cdot ]\!]_{{{\mathfrak {H}}}^2})$ and $({{\mathcal {H}}}^2,[\![\cdot ,\cdot ]\!]_{{{\mathcal {H}}}^2})$ with the inner products

$$\begin{aligned} {[}\![{{\widehat{f}} },{{\widehat{f}} }]\!]_{{{\mathfrak {H}}}^2}= -i\left( [f',f]_{{\mathfrak {H}}}-[f,f']_{{\mathfrak {H}}}\right) ,\quad {{\widehat{f}} }= \begin{pmatrix} f\\ f' \end{pmatrix}\in {{\mathfrak {H}}}^2, \\ {[}\![{{\widehat{u}} },{{\widehat{u}} }]\!]_{{{\mathcal {H}}}^2}= -i\left( (u',u)_{{\mathcal {H}}}-(u,u')_{{\mathcal {H}}}\right) ,\quad {{\widehat{u}} }= \begin{pmatrix} u\\ u' \end{pmatrix}\in {{\mathcal {H}}}^2. \end{aligned}$$

Then the Cayley transform determines the unitary operator from the Kreĭn space $({{\mathfrak {H}}}^2,[\![\cdot ,\cdot ]\!]_{{{\mathfrak {H}}}^2})$ to the Kreĭn space $({{\mathfrak {H}}}^2,J_{{{\mathfrak {H}}}^2})$ defined in Sect. 3.1,

$$\begin{aligned} C=\frac{1}{\sqrt{2}}\begin{pmatrix} i&{}1 \\ -i&{}1 \end{pmatrix}:({{\mathfrak {H}}}^2,[\![\cdot ,\cdot ]\!]_{{{\mathfrak {H}}}^2})\rightarrow ({{\mathfrak {H}}}^2,J_{{{\mathfrak {H}}}^2}) \end{aligned}$$

and similarly with ${{\mathfrak {L}}}={{\mathfrak {L}}}_1\times {{\mathfrak {L}}}_2$ it determines a unitary map** from the Kreĭn space $({{\mathcal {H}}}^2,[\![\cdot ,\cdot ]\!]_{{{\mathcal {H}}}^2})$ to the Kreĭn space $({{\mathfrak {L}}},J_{{{\mathfrak {L}}}})$,

$$\begin{aligned} C=\frac{1}{\sqrt{2}}\begin{pmatrix} i&{}1 \\ -i&{}1 \end{pmatrix}:({{\mathcal {H}}}^2,[\![\cdot ,\cdot ]\!]_{{{\mathcal {H}}}^2})\rightarrow ({{\mathfrak {L}}},J_{{{\mathfrak {L}}}}). \end{aligned}$$

It follows that the linear relation

$$\begin{aligned} {{\widetilde{\Gamma }} }=C^{-1}\circ \Gamma \circ C =\left\{ \left\{ {{\widehat{f}} }, {{\widehat{u}} }\right\} :=\left\{ \begin{pmatrix} ig'-ig\\ g'+g \end{pmatrix}, \begin{pmatrix} iv'-iv\\ v'+v \end{pmatrix}\right\} : \left\{ \begin{pmatrix} g\\ g' \end{pmatrix}, \begin{pmatrix} v\\ v' \end{pmatrix}\right\} \in \Gamma \right\} \end{aligned}$$

is unitary from $({{\mathfrak {H}}}^2,[\![\cdot ,\cdot ]\!]_{{{\mathfrak {H}}}^2})$ to $({{\mathcal {H}}}^2,[\![\cdot ,\cdot ]\!]_{{{\mathcal {H}}}^2})$ with the kernel ${\mathrm{ker\,}\,}{{\widetilde{\Gamma }} }=A$ and the domain

$$\begin{aligned} \mathrm{dom\,}{{\widetilde{\Gamma }} }=C^{-1}V_*=\left\{ {{\widehat{f}} }= \begin{pmatrix} ig'-ig\\ g'+g \end{pmatrix}: \begin{pmatrix} g\\ g' \end{pmatrix}\in V_*\right\} . \end{aligned}$$

Since the map** ${{\widetilde{\Gamma }} }$ is isometric from $({{\mathfrak {H}}}^2,[\![\cdot ,\cdot ]\!]_{{{\mathfrak {H}}}^2})$ to $({{\mathcal {H}}}^2,[\![\cdot ,\cdot ]\!]_{{{\mathcal {H}}}^2})$ the following (Green’s) identity

$$\begin{aligned}{}[f',f]_{{\mathfrak {H}}}-[f,f']_{{\mathfrak {H}}}=[u',u]_{{\mathcal {H}}}-[u,u']_{{\mathcal {H}}}\end{aligned}$$

holds for all $\{{{\widehat{f}} },{{\widehat{u}} }\}\in {{\widetilde{\Gamma }} }$ and due to [19, \({{\widetilde{\Gamma }} }\) is surjective or, equivalently, when $({{\mathcal {H}}}^2, \Gamma )$ is an ordinary boundary pair for V.

Finally the main results in this subsection are specialized to unitary boundary pairs of Hilbert space isometries. In a Hilbert space setting the properties of the $\gamma$-fields can be connected more directly to the properties of the Weyl function.

Proposition 3.27

Let $({{{\mathfrak {L}}}},\Gamma )$ be a unitary boundary pair for an isometric operator V in the Pontryagin space ${{{\mathfrak {H}}}}$. Then

$$\begin{aligned} \mathrm{mul\,}\Gamma _1={\mathrm{ker\,}\,}\gamma _1(\lambda ), \; \lambda \in {{\mathbb {D}}}, \quad \text {and}\quad \mathrm{mul\,}\Gamma _2={\mathrm{ker\,}\,}\gamma _2(\lambda ), \; \lambda \in {{\mathbb {D}}}_e, \end{aligned}$$

and $({{{\mathfrak {L}}}},\Gamma )$ admits the following further properties.

• (a) The following statements are equivalent:

  • (i) $\Gamma$ is single valued (i.e. $\mathrm{mul\,}\Gamma =\{0\}$);

  • (ii) $\mathrm{ran\,}\Gamma$ is dense in ${{\mathfrak {L}}}_1\times {{\mathfrak {L}}}_2$;

  • (iii) ${\mathrm{ker\,}\,}\gamma _1(\lambda )=\{0\}$ for some (equivalently for all) $\lambda \in {{\mathbb {D}}}$;

  • (iv) ${\mathrm{ker\,}\,}\gamma _2(\lambda )=\{0\}$ for some (equivalently for all) $\lambda \in {{\mathbb {D}}}_e$;

  • (v) ${\mathrm{ker\,}\,}{{\textsf {S}}}^\Theta _\lambda (\lambda )=\{0\}$ for some (equivalently for all) $\lambda \in {{\mathbb {D}}}\cup {{\mathbb {D}}}_e$.

• (b) Moreover, the following statements are equivalent:

  • (i) $({{{\mathfrak {L}}}},\Gamma )$ reduces to an ordinary boundary triple $({{{\mathfrak {L}}}},\Gamma _1,\Gamma _2)$ for V;

  • (ii) $\mathrm{dom\,}\Gamma =V^{-[*]}$ and $\mathrm{mul\,}\Gamma =\{0\}$;

  • (iii) $\mathrm{ran\,}\Gamma ={{\mathfrak {L}}}_1\times {{\mathfrak {L}}}_2$;

  • (iv) ${\mathrm{ker\,}\,}\gamma _1(\lambda )=\{0\}$ and $\mathrm{ran\,}\gamma _1(\lambda )$ is closed for some (equivalently for all) $\lambda \in {{\mathbb {D}}}$;

  • (v) ${\mathrm{ker\,}\,}\gamma _2(\lambda )=\{0\}$ and $\mathrm{ran\,}\gamma _2(\lambda )$ is closed for some (equivalently for all) $\lambda \in {{\mathbb {D}}}_e$;

  • (vi) $0\in \rho ({{\textsf {S}}}^\Theta _\lambda (\lambda ))$, i.e., $\Vert \Theta (\lambda )\Vert <1$ for some (equivalently for all) $\lambda \in {{\mathbb {D}}}\cup {{\mathbb {D}}}_e$.

Proof

(a) The equivalences (i)–(iv) follow from Proposition 3.23; see also Theorem 3.19. To see the equivalence with item (v) apply Proposition 3.22 with $\lambda =\mu$:

$$\begin{aligned} {{\textsf {S}}}^\Theta _\lambda (\lambda )=\frac{I-\Theta (\lambda )^* \Theta (\lambda )}{1-|\lambda |^2}=\gamma _1(\lambda )^*\gamma _1(\lambda )\ge 0, \quad \lambda \in {{\mathbb {D}}}; \end{aligned}$$

(3.77)

$$\begin{aligned} {{\textsf {S}}}^\Theta _\lambda (\lambda )=\frac{I-\Theta ^\#(\lambda )^* \Theta ^\#(\lambda )}{1-|\lambda |^2}=-\gamma _2(\lambda )^*\gamma _2(\lambda )\le 0, \quad \lambda \in {{\mathbb {D}}}_e. \end{aligned}$$

(3.78)

In the present Hilbert space case these identities lead to

$$\begin{aligned} {\mathrm{ker\,}\,}{{\textsf {S}}}^\Theta _\lambda (\lambda )={\mathrm{ker\,}\,}\gamma _1(\lambda ), \; \lambda \in {{\mathbb {D}}}; \quad {\mathrm{ker\,}\,}{{\textsf {S}}}^\Theta _\lambda (\lambda )={\mathrm{ker\,}\,}\gamma _2(\lambda ),\; \lambda \in {{\mathbb {D}}}_e. \end{aligned}$$

This implies the equivalence of (iii), (iv) and (v) in part (a).

(b) Here the equivalence of (i) and (ii) holds just by the definition of an ordinary boundary triple (see [41]). The equivalences (ii)–(v) are obtained from Proposition 3.24 (cf. also Theorem 3.19). To see the equivalence with item (vi) apply the identities (3.77), (3.78):

$$\begin{aligned} \begin{array}{ll} 0\in \rho (I-\Theta (\lambda )^* \Theta (\lambda )) &{}\Longleftrightarrow \quad 0\in \rho (\gamma _1(\lambda )^*\gamma _1(\lambda )), \quad \lambda \in {{\mathbb {D}}}; \\ 0\in \rho (I-\Theta ^\#(\lambda )^* \Theta ^\#(\lambda )) &{}\Longleftrightarrow \quad 0\in \rho (\gamma _2(\lambda )^*\gamma _2(\lambda )),\quad \lambda \in {{\mathbb {D}}}_e. \end{array} \end{aligned}$$

Thus $I-\Theta (\lambda )^* \Theta (\lambda )$ and $I-\Theta ^\#(\lambda )^* \Theta ^\#(\lambda )$ are uniformly positive or, equivalently, $\Vert \Theta (\lambda )\Vert <1$ and $\Vert \Theta ^\#(\lambda ))\Vert <1$. This completes the proof. $\square$

Notice that part (b) of Proposition 3.27 contains the properties and generality that can be attained when applying (ordinary) boundary triples for isometric operators which have been introduced and studied in [41, 42].

Remark 3.28

An analog of boundary triple in scattering form (3.2) is encountered in [17], where extension theory of multiplication operators in indefinite de Branges spaces was developed. The role of the Weyl function in that work is played by de Branges matrix.

4 Extension Theory and Generalized Coresolvents

An extension ${{\widetilde{V}} }$ of the isometric operator V is called proper, if $V\subsetneq {{\widetilde{V}} }\subsetneq V^{-[*]}$. The set of proper extensions of V was parametrized in [5] via an ordinary boundary triple. In the present section we consider extensions ${{\widetilde{V}} }$ of the isometric operator V, which are proper with respect to a given unitary boundary pair $({{\mathfrak {L}}},\Gamma )$, i.e.

$$\begin{aligned} V\subsetneq {{\widetilde{V}} }\subsetneq V_*=\mathrm{dom\,}\Gamma . \end{aligned}$$

(4.1)

For such extensions we prove sufficient conditions for regularity of a point $\lambda$, find formulas for their coresolvents and then apply them for a description of generalized coresolvents of the isometric operator V.

4.1 A Preparatory Lemma

Lemma 4.1

Let $V:{{{\mathfrak {H}}}}\rightarrow {{{\mathfrak {H}}}}$ be an isometric operator and let $\Pi =({{{\mathfrak {L}}}},\Gamma )$ be a unitary boundary pair for V. Then:

• (i) For every $\begin{pmatrix} f_1\\ f_1' \end{pmatrix}\in V_1$ and $\lambda \in {{\mathcal {D}}}$, as defined in (3.48) one has

  $$\begin{aligned} \left\{ \begin{pmatrix} f_1\\ f_1' \end{pmatrix},\frac{1}{\lambda }\gamma _2^\#(\lambda )(f_1'-\lambda f_1)\right\} \in \Gamma _2. \end{aligned}$$

  (4.2)

  If $0\in {{\mathcal {D}}}$ then the formula (4.2) for $\lambda =0$ takes the form

  $$\begin{aligned} \left\{ \begin{pmatrix} f_1\\ f_1' \end{pmatrix},(\gamma _2^\#)'(0)f_1\right\} \in \Gamma _2. \end{aligned}$$

  (4.3)

• (ii) For every $\begin{pmatrix} f_2\\ f_2' \end{pmatrix}\in V_2$ and $\lambda \in {{\mathcal {D}}}_e$ one has

  $$\begin{aligned} \left\{ \begin{pmatrix} f_2\\ f_2' \end{pmatrix},-\frac{1}{\lambda }\gamma _1^\#(\lambda )(f_1'-\lambda f_1)\right\} \in \Gamma _1. \end{aligned}$$

  (4.4)

Proof

(i) Using Definitions 3.8, 3.9 it is seen that

$$\begin{aligned} \left\{ \begin{pmatrix} \gamma _2\left(\frac{1}{\overline{\lambda }}\right)u_2 \\ \frac{1}{\overline{\lambda }}\gamma _2\left(\frac{1}{\overline{\lambda }}\right)u_2 \end{pmatrix}, \begin{pmatrix} \Theta \left(\frac{1}{\overline{\lambda }}\right)^*u_2 \\ u_2 \end{pmatrix} \right\} ,\quad \left\{ \begin{pmatrix} f_1 \\ f_1' \end{pmatrix}, \begin{pmatrix} 0 \\ v_2 \end{pmatrix} \right\} \in \Gamma , \end{aligned}$$

(4.5)

for all $u_2\in {{\mathfrak {L}}}_2$, and some $v_2\in {{\mathfrak {L}}}_2$, $\lambda \in {{\mathcal {D}}}{\setminus }\{0\}$. By applying the identity (3.2) to these elements one obtains

$$\begin{aligned} \left[\gamma _2\left(\frac{1}{\overline{\lambda }}\right)u_2,f_1\right]_{{\mathfrak {H}}}-\frac{1}{\overline{\lambda }}\left[\gamma _2\left(\frac{1}{\overline{\lambda }}\right)u_2,f_1'\right]_{{\mathfrak {H}}}=-(u_2, v_2)_{{{\mathfrak {L}}}_2}, \end{aligned}$$

or, equivalently,

$$\begin{aligned} \left(u_2,\gamma _2\left(\frac{1}{\overline{\lambda }}\right)^*(\frac{1}{\lambda }f_1'-f_1)\right)_{{{\mathfrak {L}}}_2}=(u_2, v_2)_{{{\mathfrak {L}}}_2}. \end{aligned}$$

Since $u_2\in {{\mathfrak {L}}}_2$ is arbitrary this implies the equality

$$\begin{aligned} v_2=\frac{1}{\lambda }\gamma _2^\#\left(\lambda \right)\left(f_1'-\lambda f_1\right) \end{aligned}$$

which in combination with (4.5) yields (4.2).

The equality (4.3) is implied by (4.2) and (3.49).

(ii) Similarly, applying (3.2) to the vectors

$$\begin{aligned} \left\{ \begin{pmatrix} \gamma _1\left(\frac{1}{\overline{\lambda }}\right)u_1 \\ \frac{1}{\overline{\lambda }}\gamma _1\left(\frac{1}{\overline{\lambda }}\right)u_1 \end{pmatrix}, \begin{pmatrix} u_1 \\ \Theta \left(\frac{1}{\overline{\lambda }}\right)u_1 \end{pmatrix} \right\} ,\, \left\{ \begin{pmatrix} f_2 \\ f_2' \end{pmatrix}, \begin{pmatrix} v_1\\ 0 \end{pmatrix} \right\} \in \Gamma , \end{aligned}$$

(4.6)

where $u_1, v_1 \in {{\mathfrak {L}}}_1$, $f_2,f_2'\in {{\mathfrak {H}}}$, $\lambda \in {{\mathcal {D}}}_e$, one obtains

$$\begin{aligned} \left[\gamma _1\left(\frac{1}{\overline{\lambda }}\right)u_1,f_2\right]_{{\mathfrak {H}}}-\frac{1}{\overline{\lambda }}\left[\gamma _1\left(\frac{1}{\overline{\lambda }}\right)u_1,f_2'\right]_{{\mathfrak {H}}}=(u_1, v_1)_{{{\mathfrak {L}}}_1} \end{aligned}$$

and

$$\begin{aligned} \left(u_1,\gamma _1^\#\left(\lambda \right)\left(f_2-\frac{1}{\lambda }f_2'\right)\right)_{{{\mathfrak {L}}}_1}=(u_1, v_1)_{{{\mathfrak {L}}}_1}. \end{aligned}$$

This implies

$$\begin{aligned} v_1=-\frac{1}{\lambda }\gamma _1^\#\left(\lambda \right)\left(f_2'-\lambda f_2\right), \end{aligned}$$

which together with (4.6) yields (4.4). $\square$

4.2 Weyl Function and Spectrum of Proper Extensions of V

A unitary boundary pair $({{\mathfrak {L}}},\Gamma )$ is a tool which allows to determine those extensions ${{\widetilde{V}} }$ of V that satisfy $V\subsetneq {{\widetilde{V}} }\subsetneq V_*$ in the following way. Let $\Phi$ be a linear relation from ${{\mathfrak {L}}}_1$ to ${{\mathfrak {L}}}_2$ represented in the form

$$\begin{aligned} \Phi =\left\{ \left( {\begin{array}{c}\Phi _1h\\ \Phi _2h\end{array}}\right) :\,h\in {{\mathcal {H}}}\right\} , \end{aligned}$$

(4.7)

where ${{\mathcal {H}}}$ is an auxiliary Hilbert space and $\Phi _j$ are bounded linear operators $\Phi _j:{{\mathcal {H}}}\rightarrow {{\mathfrak {L}}}_j$ $(j=1,2)$, such that

$$\begin{aligned} {\mathrm{ker\,}\,}(\Phi _1^*\Phi _1+\Phi _2^*\Phi _2)=\{0\}. \end{aligned}$$

(4.8)

Then $\Phi$ is closed if and only if

$$\begin{aligned} 0\in \rho (\Phi _1^*\Phi _1+\Phi _2^*\Phi _2). \end{aligned}$$

(4.9)

Associate with $\Phi$ an extension $V_\Phi$ of V by

$$\begin{aligned} V_\Phi = \left\{ \begin{pmatrix} f \\ f' \end{pmatrix} \in V_*:\left\{ \begin{pmatrix} f \\ f' \end{pmatrix}, \begin{pmatrix} \Phi _1h \\ \Phi _2 h \end{pmatrix}\right\} \in \Gamma \text { for some } h\in {{\mathcal {H}}}\right\} . \end{aligned}$$

(4.10)

The following theorem gives a description of the spectrum of $V_\Phi$ and contains a Kreĭn type resolvent formula.

Theorem 4.2

Let V be a closed isometric operator in ${{\mathfrak {H}}}$, let $\Pi =({{{\mathfrak {L}}}},\Gamma )$ be a unitary boundary pair for V, let $\Phi \in {\mathbf {B}}({{\mathcal {H}}},{{\mathfrak {L}}})$ and let (4.8) hold. If $\lambda \in {{\mathcal {D}}}$ then:

• (i) $\lambda \in \sigma _p(V_\Phi ) \Longrightarrow {\mathrm{ker\,}\,}(\Phi _2-\Theta (\lambda )\Phi _1)\ne \{0\}$;

• (ii) $\Phi _2-\Theta (\lambda )\Phi _1:{{\mathcal {H}}}\rightarrow {{\mathfrak {L}}}_2$ has a bounded inverse $\Longrightarrow \lambda \in \rho (V_\Phi )$.

When (ii) is satisfied the resolvent of $V_\Phi$ takes the form

$$\begin{aligned} (V_\Phi -\lambda I_{{{\mathfrak {H}}}})^{-1}=(V_1 - \lambda I_{{{\mathfrak {H}}}})^{-1}+\frac{1}{\lambda }\gamma _1(\lambda )\Phi _1(\Phi _2-\Theta (\lambda )\Phi _1)^{-1}\gamma _2^\#(\lambda )\quad (\lambda \in {{\mathcal {D}}}).\nonumber \\ \end{aligned}$$

(4.11)

If $\lambda \in {{\mathcal {D}}}_e$ then:

• (iii) $\lambda \in \sigma _p(V_\Phi ) \Longrightarrow {\mathrm{ker\,}\,}(\Phi _1-\Theta ^\#(\lambda )\Phi _2)\ne \{0\}$;

• (iv) $\Phi _1-\Theta ^\#(\lambda )\Phi _2:{{\mathcal {H}}}\rightarrow {{\mathfrak {L}}}_1$ has a bounded inverse $\Longrightarrow \lambda \in \rho (V_\Phi )$;

When (iv) is satisfied the resolvent of $V_\Phi$ takes the form

$$\begin{aligned} (V_\Phi -\lambda I_{{{\mathfrak {H}}}})^{-1}=(V_2-\lambda I_{{{\mathfrak {H}}}})^{-1}-\frac{1}{\lambda }\gamma _2(\lambda )\Phi _2(\Phi _1-\Theta ^\#(\lambda )\Phi _2)^{-1}\gamma _1^\#(\lambda )\quad (\lambda \in {{\mathcal {D}}}_e).\nonumber \\ \end{aligned}$$

(4.12)

If, in addition, $\Pi =({{{\mathfrak {L}}}},\Gamma )$ is an ordinary boundary triple for V then the implications (i)–(iv) become equivalences.

Proof

The proof is divided into steps.

1. Verification of (i). If $\lambda \in {{\mathcal {D}}}\cap \sigma _p(V_\Phi )$, ${{\mathcal {D}}}=\rho (V_1)$, then there is $f\in {{\mathfrak {H}}}\setminus \{0\}$ and $h\in {{\mathcal {H}}}$ such that

$$\begin{aligned} \left\{ \begin{pmatrix} f \\ \lambda f \end{pmatrix}, \begin{pmatrix} \Phi _1h \\ \Phi _2h \end{pmatrix}\right\} \in \Gamma . \end{aligned}$$

(4.13)

Since $\lambda \in {{\mathcal {D}}}$, one has $\lambda \not \in \sigma _p(V)$ and hence $h\ne 0$. By Definition 3.9 this means that

$$\begin{aligned} \Theta (\lambda )\Phi _1h=\Phi _2h, \quad \lambda \in {{\mathcal {D}}}. \end{aligned}$$

(4.14)

Hence, $h\in {\mathrm{ker\,}\,}(\Phi _2-\Theta (\lambda )\Phi _1)$.

2. Verification of (ii). First assume that $(\Phi _2-\Theta (\lambda )\Phi _1)$ has a bounded inverse with $\lambda \in {{\mathcal {D}}}\setminus \{0\}$. Let us find a solution ${{\widehat{f}} }= \begin{pmatrix} f \\ f' \end{pmatrix} \in V_\Phi$ of the equation

$$\begin{aligned} f'-\lambda f=g \end{aligned}$$

(4.15)

for arbitrary $g\in {{\mathfrak {H}}}$. Since $\lambda \in {{\mathcal {D}}}=\rho (V_1)$, there are $f_1,f_1'\in {{\mathfrak {H}}}$, such that

$$\begin{aligned} f_1'-\lambda f_1=g\quad and \quad \begin{pmatrix} f_1 \\ f'_1 \end{pmatrix}\in V_1. \end{aligned}$$

Hence

$$\begin{aligned} f_1=(V_1-\lambda I_{{{\mathfrak {H}}}})^{-1}g. \end{aligned}$$

(4.16)

By Lemma 4.1

$$\begin{aligned} \left\{ \begin{pmatrix} f_1 \\ f_1' \end{pmatrix}, \begin{pmatrix} 0\\ u_2 \end{pmatrix}\right\} \in \Gamma ,\;where \quad u_2= \frac{1}{\lambda }\gamma ^\#_2(\lambda )g=\frac{1}{\lambda }\gamma ^\#_2(\lambda )(f_1'-\lambda f_1). \end{aligned}$$

(4.17)

Now choose $h=(\Phi _2-\Theta (\lambda )\Phi _1)^{-1}u_2$ and apply (3.50) to get

$$\begin{aligned} \left\{ \begin{pmatrix} \gamma _2(\lambda )\Phi _1h \\ \lambda \gamma _2(\lambda )\Phi _1h \end{pmatrix}, \begin{pmatrix} \Phi _1h \\ \Theta (\lambda )\Phi _1h \end{pmatrix} \right\} \in \Gamma . \end{aligned}$$

(4.18)

Combining (4.17) and (4.18) one obtains

$$\begin{aligned} \left\{ \begin{pmatrix} f_1 \\ f_1' \end{pmatrix} +\begin{pmatrix} \gamma _1(\lambda )\Phi _1h \\ \lambda \gamma _1(\lambda )\Phi _1h \end{pmatrix} , \begin{pmatrix} \Phi _1h \\ u_2 +\Theta (\lambda )\Phi _1h \end{pmatrix}\right\} \in \Gamma . \end{aligned}$$

(4.19)

Setting

$$\begin{aligned} \begin{pmatrix} f \\ f' \end{pmatrix}= \begin{pmatrix} f_1 \\ f'_1 \end{pmatrix}+ \begin{pmatrix} \gamma _1(\lambda ))\Phi _1h \\ \lambda \gamma _1(\lambda ))\Phi _1h \end{pmatrix} \end{aligned}$$

(4.20)

and using the equality

$$\begin{aligned} u_2+\Theta (\lambda )\Phi _1h=(I+\Theta (\lambda )\Phi _1(\Phi _2-\Theta (\lambda )\Phi _1)^{-1})u_2=\Phi _2h \end{aligned}$$

one obtains from (4.19)

$$\begin{aligned} \left\{ \begin{pmatrix} f \\ f' \end{pmatrix}, \begin{pmatrix} \Phi _1h \\ \Phi _2h \end{pmatrix}\right\} \in \Gamma . \end{aligned}$$

(4.21)

Therefore, the equation (4.15) has a solution ${{\widehat{f}} }= \begin{pmatrix} f \\ f' \end{pmatrix} \in V_\Phi$ and

$$\begin{aligned} \begin{aligned} f&= f_1+ \gamma _1(\lambda )\Phi _1h \\&= (V_1-\lambda I_{{{\mathfrak {H}}}})^{-1}g + \frac{1}{\lambda }\gamma _1(\lambda )\Phi _1(\Phi _2-\Theta (\lambda )\Phi _1)^{-1} \gamma _2^\#(\lambda )g. \end{aligned} \end{aligned}$$

Next assume that $0\in {{\mathcal {D}}}$ and $0\in \rho (\Phi _2-\Theta (0)\Phi _1)$. Then by Lemma 4.1 the equality (4.17) holds with $u_2=(\gamma _2^\#)'(0)f_1$ and now combining this analog of (4.17) with (4.18) yields (4.21). The formula (4.11) for $\lambda =0$ takes the form

$$\begin{aligned} V_\Phi ^{-1}= V_1^{-1} + \gamma _1(0)\Phi _1(\Phi _2-\Theta (0)\Phi _1)^{-1} (\gamma _2^\#)'(0). \end{aligned}$$

This completes the proof of the implication in (ii) and the formula (4.11).

3. Verification of (iii). If $\lambda \in \sigma _p(V_\Phi )\cap {{\mathcal {D}}}_e$ then there is $f\in {{\mathfrak {H}}}\backslash \{0\}$ and $h\in {{\mathcal {H}}}$ such that (4.13) holds. Again, since $\lambda \in {{\mathcal {D}}}$, one has $\lambda \not \in \sigma _p(V)$ and hence $h\ne 0$. By Definition 3.9 one gets

$$\begin{aligned} \Phi _1h-\Theta ^\#(\lambda )\Phi _2h=0. \end{aligned}$$

Hence $h\in {\mathrm{ker\,}\,}(\Phi _1-\Theta ^\#(\lambda )\Phi _2)$.

4. Verification of (iv). Assume that $\Phi _1-\Theta ^\#(\lambda )\Phi _2$ has a bounded inverse and $\lambda \in {{\mathcal {D}}}_e$. Since $\lambda \in {{\mathcal {D}}}_e=\rho (V_2)$, there are $f_2,f_2'\in {{\mathfrak {H}}}$, such that

$$\begin{aligned} f_2'-\lambda f_2=g\quad and \quad \begin{pmatrix} f_2 \\ f'_2 \end{pmatrix}\in V_2. \end{aligned}$$

Hence $f_2=(V_2-\lambda I_{{{\mathfrak {H}}}})^{-1}g$. By Lemma 4.1

$$\begin{aligned} \left\{ \begin{pmatrix} f_2 \\ f_2' \end{pmatrix}, \begin{pmatrix} u_1\\ 0 \end{pmatrix}\right\} \in \Gamma \quad with \quad u_1= -\frac{1}{\lambda }\gamma ^\#_1(\lambda )g. \end{aligned}$$

(4.22)

Now choose $h=(\Phi _1-\Theta ^\#(\lambda )\Phi _2)^{-1}u_1$ and apply (3.51) to get

$$\begin{aligned} \left\{ \begin{pmatrix} \gamma _2(\lambda )\Phi _2h \\ \lambda \gamma _2(\lambda )\Phi _2h \end{pmatrix}, \begin{pmatrix} \Theta ^\#(\lambda )\Phi _2h \\ \Phi _2h \end{pmatrix} \right\} \in \Gamma . \end{aligned}$$

(4.23)

Combining (4.22) and (4.23) one obtains

$$\begin{aligned} \left\{ \begin{pmatrix} f \\ f' \end{pmatrix}, \begin{pmatrix} \Phi _1h \\ \Phi _2h \end{pmatrix}\right\} \in \Gamma , \;where \quad \begin{pmatrix} f \\ f' \end{pmatrix}=\begin{pmatrix} f_2 \\ f'_2 \end{pmatrix}+ \begin{pmatrix} \gamma _2(\lambda )\Phi _2h \\ \lambda \gamma _2(\lambda )\Phi _2h \end{pmatrix}. \end{aligned}$$

(4.24)

Making use of (4.24), (4.22), and the above formulas for h and $f_2$ one obtains

$$\begin{aligned} \begin{aligned} f&= f_2+ \gamma _2(\lambda )\Phi _2h \\&= (V_2-\lambda I_{{{\mathfrak {H}}}})^{-1}g - \frac{1}{\lambda }\gamma _2(\lambda )\Phi _2(\Phi _1- \Theta ^\#(\lambda )\Phi _2)^{-1} \gamma _1^\#(\lambda )g. \end{aligned} \end{aligned}$$

This proves (4.12) and the implication in (iv).

5. Verification of the reverse implication in (i) for the case of an ordinary boundary triple $({{{\mathfrak {L}}}},\Gamma )$. Let $(\Phi _2-\Theta (\lambda )\Phi _1)h=0$ for some $h\in {{\mathcal {H}}}\backslash \{0\}$. Then it follows from

$$\begin{aligned} \left\{ \begin{pmatrix} \gamma _1(\lambda )\Phi _1h \\ \lambda \gamma _1(\lambda )\Phi _1h \end{pmatrix}, \begin{pmatrix} \Phi _1h \\ \Theta (\lambda )\Phi _1h \end{pmatrix} \right\} \in \Gamma \end{aligned}$$

(4.25)

and (4.14) that (4.13) holds with $f=\gamma _1(\lambda )\Phi _1h$. Notice that $\Phi _1h\ne 0$ since otherwise $\Phi _2h= 0$ by (4.14), which contradicts to (4.8). Therefore, $f\ne 0$ since ${\mathrm{ker\,}\,}\gamma _1(\lambda )=\{0\}$ for the ordinary boundary triple $\Pi =({{{\mathfrak {L}}}},\Gamma )$; see Proposition 3.24. Thus $\lambda \in \sigma _p(V_\Phi )$.

6. Verification of the reverse implication in (ii) for the case of an ordinary boundary triple $({{{\mathfrak {L}}}},\Gamma )$. Let $\lambda \in \rho (V_\Phi )$. By virtue of item 5 to prove the boundedness of the inverse $(\Phi _2-\Theta (\lambda )\Phi _1)^{-1}:{{\mathfrak {L}}}_2\rightarrow {{\mathcal {H}}}$ it is enough to show that

$$\begin{aligned} \mathrm{ran\,}(\Phi _2-\Theta (\lambda )\Phi _1)={{\mathfrak {L}}}_2. \end{aligned}$$

(4.26)

By assumption $\lambda \in \rho (V_\Phi )\cup {{\mathcal {D}}}$ and hence for arbitrary $g\in {{\mathfrak {H}}}$ one can find vectors $\begin{pmatrix} f\\ f' \end{pmatrix}\in V_\Phi$ and $\begin{pmatrix} f_1\\ f_1' \end{pmatrix}\in V_1$ such that

$$\begin{aligned} f_1'-\lambda f_1=f-\lambda f'=g\quad (\lambda \in \rho (V_\Phi )\cup {{\mathcal {D}}}). \end{aligned}$$

Then (4.17)–(4.19) hold for some $h\in {{\mathcal {H}}}$ and, in particular,

$$\begin{aligned} (\Phi _2-\Theta (\lambda )\Phi _1)h=u_2=\frac{1}{\lambda }\gamma _2^\#(\lambda )g. \end{aligned}$$

Since $g\in {{\mathfrak {H}}}$ is arbitrary and for an ordinary boundary triple $\mathrm{ran\,}\gamma _2^\#(\lambda )={{\mathfrak {L}}}_2$, the claim (4.26) is proved. By Open Map** Theorem the operator $T:=\Phi _2-\Theta (\lambda )\Phi _1:{{\mathcal {H}}}\rightarrow {{\mathfrak {L}}}_2$ has a bounded inverse, since $T\in {{\mathbf {B}}}({{\mathcal {H}}},{{\mathfrak {L}}}_2)$, ${\mathrm{ker\,}\,}T=\{0\}$ and $\mathrm{ran\,}T={{\mathfrak {L}}}_2$.

$\square$

The operator function $(I_{{{\mathfrak {H}}}}-z V_\Phi )^{-1}$ is called the coresolvent of $V_\Phi$. Setting $\lambda = 1/z$ in Theorem 4.2 one obtains the following statement for coresolvents of $V_\Phi$.

Corollary 4.3

Let V be a closed isometric operator in ${{\mathfrak {H}}}$, let $\Pi =({{{\mathfrak {L}}}},\Gamma )$ be a unitary boundary pair for V, let $\Phi \in {\mathbf {B}}({{\mathcal {H}}},{{\mathfrak {L}}})$ and let (4.8) hold. If $z\in \overline{{{\mathcal {D}}}}$ then:

• (i) $z\in \sigma _p(V_\Phi ^{-1}) \Longrightarrow {\mathrm{ker\,}\,}\left( \Phi _1-\Theta \left( {\bar{z}}\right) ^*\Phi _2\right) \ne \{0\}$;

• (ii) $\Phi _1-\Theta ({\bar{z}})^*\Phi _2:{{\mathcal {H}}}\rightarrow {{\mathfrak {L}}}_1$ has a bounded inverse $\Longrightarrow z\in \rho (V_\Phi ^{-1})$.

When (ii) is satisfied then the coresolvent of $V_\Phi$ takes the form

$$\begin{aligned} (I_{{{\mathfrak {H}}}}-z V_\Phi )^{-1}=(I_{{{\mathfrak {H}}}}-z V_2)^{-1}+\gamma _2\left( \frac{1}{z}\right) \Phi _2\left( \Phi _1-\Theta ({\bar{z}})^*\Phi _2\right) ^{-1}\gamma _1({\bar{z}})^*.\nonumber \\ \end{aligned}$$

(4.27)

If $z\in \overline{{{\mathcal {D}}}_e}$ then:

• (iii) $z\in \sigma _p(V_\Phi ^{-1}) \Longrightarrow {\mathrm{ker\,}\,}\left( \Phi _2-\Theta \left( {1}/{z}\right) \Phi _1\right) \ne \{0\}$;

• (iv) $\Phi _2-\Theta \left( {1}/{z}\right) \Phi _1:{{\mathcal {H}}}\rightarrow {{\mathfrak {L}}}_2$ has a bounded inverse $\Longrightarrow z\in \rho (V_\Phi ^{-1})$.

When (iv) is satisfied then the coresolvent of $V_\Phi$ takes the form

$$\begin{aligned} (I_{{{\mathfrak {H}}}}-z V_\Phi )^{-1}=(I_{{{\mathfrak {H}}}} - z V_1)^{-1}-\gamma _1\left( {1}/{z}\right) \Phi _1\left( \Phi _2-\Theta \left( {1}/{z}\right) \Phi _1\right) ^{-1}\gamma _2({\bar{z}})^*.\nonumber \\ \end{aligned}$$

(4.28)

If, in addition, $\Pi =({{{\mathfrak {L}}}},\Gamma )$ is an ordinary boundary triple for V then the implications (i)–(iv) become equivalences.

Remark 4.4

If $({{\mathfrak {L}}}_1\oplus {{\mathfrak {L}}}_2,\Gamma _1,\Gamma _2)$ is an ordinary boundary triple for V then every closed proper extension of V can be represented in the form (4.10) with $\Phi _j\in {\mathbf {B}}({{\mathcal {H}}},{{\mathfrak {L}}}_j)$ $(j=1,2)$ such that (4.9) holds due to [5, Theorem 2.1], [27, Proposition 6.12]. Moreover, if $\Phi$ is defined by (4.7), then the following equivalences hold:

• (1) $V_\Phi$ is an isometric relation in ${{\mathcal {H}}}$ $\Longleftrightarrow$ $\Phi$ is the graph of an isometric operator;

• (2) $V_\Phi$ is a unitary relation in ${{\mathcal {H}}}$ $\Longleftrightarrow$ $\Phi$ is the graph of a unitary operator;

• (3) $V_\Phi$ is a contractive relation in ${{\mathcal {H}}}$ $\Longleftrightarrow$ $\Phi$ is the graph of a contraction.

The fact that the implications (i)–(iv) of Theorem 4.2 become equivalences for an ordinary boundary triple $\Pi =({{{\mathfrak {L}}}},\Gamma )=({{{\mathfrak {L}}}},\Gamma _1,\Gamma _2)$ was proved in [41] in the case when $\kappa =0$, and in [5] in the case $\kappa \ne 0$.

4.3 Description of Generalized Coresolvents

Definition 4.5

(see [38, 39]) An operator-valued function ${\mathbf {K}}_\lambda$ holomorphic in a domain ${{\mathcal {O}}}\subseteq {{\mathbb {D}}}$ with values in ${\mathbf {B}}({{\mathfrak {H}}})$ is called a generalized coresolvent of an isometric operator $V:{{\mathfrak {H}}}\rightarrow {{\mathfrak {H}}}$, if there exist a Pontryagin space ${{\widetilde{{{\mathfrak {H}}}}} }\supset {{{\mathfrak {H}}}}$ with negative index ${{\widetilde{\kappa }} }=\kappa _-({{\widetilde{{\mathfrak {H}}}} })$ and a unitary extension ${{\widetilde{V}} }:{{\widetilde{{\mathfrak {H}}}} }\rightarrow {{\widetilde{{\mathfrak {H}}}} }$ of the operator V such that ${{\mathcal {O}}}\subseteq \rho ({{\widetilde{V}} }^{-1})$, and

$$\begin{aligned} {\mathbf {K}}_{z}=P_{{{{\mathfrak {H}}}}}\left(I_{{{\widetilde{{\mathfrak {H}}}} }}-{z}{{\widetilde{V}} }\right)^{-1}\upharpoonright {{{\mathfrak {H}}}}, \quad {z}\in {{\mathcal {O}}}, \end{aligned}$$

(4.29)

where $P_{{{{\mathfrak {H}}}}}$ is the orthogonal projection from ${{\widetilde{{{\mathfrak {H}}}}} }$ onto ${{{\mathfrak {H}}}}$. Notice that in [38] the operator function ${\mathbf {K}}_{z}$ in (4.29) is called a generalized resolvent of V.

The representation (4.29) of the generalized coresolvent of V is called minimal, if

$$\begin{aligned} {{\widetilde{{{\mathfrak {H}}}}} }=\overline{\text{ span }}\left\{ {{\mathfrak {H}}}+(I_{{{\widetilde{{\mathfrak {H}}}} }}-{z}{{\widetilde{V}} })^{-1}{{\mathfrak {H}}}:\,\lambda \in {{\mathcal {O}}}\right\} . \end{aligned}$$

The generalized coresolvent ${\mathbf {K}}_{z}$ is said to be k-regular, if $k=\kappa _-({{\widetilde{{\mathfrak {H}}}} }[-]{{\mathfrak {H}}})$ for a minimal representation (4.29).

Every generalized coresolvent ${\mathbf {K}}_{z}$ of the isometric operator V admits a minimal representation (4.29) and every two minimal representations of ${\mathbf {K}}_{z}$ are unitarily equivalent, see [14, Proposition 4.1] for the case of a symmetric operator.

Lemma 4.6

Let ${{\mathbf {K}}}_{z}$ be a $({{\widetilde{\kappa }} }-\kappa )$-regular generalized coresolvent. Then the kernel

$$\begin{aligned} {\textsf {R}}_{w}({z}):=\frac{{{\mathbf {K}}}_{z}+{{\mathbf {K}}}_{w}^{[*]}-I}{1-{z} \overline{w}}-{{\mathbf {K}}}_{w}^{[*]}{{\mathbf {K}}}_{z} \end{aligned}$$

(4.30)

has ${{\widetilde{\kappa }} }-\kappa$ negative squares on $\rho ({{\widetilde{V}} }^{-1})\cap {{\mathbb {D}}}$.

Proof

Let $\{{z}_j\}_{j=1}^n$ be a set of points in $\rho ({{\widetilde{V}} }^{-1})\cap {{\mathbb {D}}}$ and let $g_j \in {{\mathfrak {H}}}$, $j=1,2,\dots ,n$. Denote

$$\begin{aligned} {{\widetilde{f}} }_j:= (I_{{{\widetilde{{\mathfrak {H}}}} }}-{z}_j{{\widetilde{V}} })^{-1}g_j,\quad f_j:= P_{{\mathfrak {H}}}{{\widetilde{f}} }_j={{\mathbf {K}}}_{{z}_j}g_j,\quad (j=1,2,\dots ,n). \end{aligned}$$

(4.31)

Then it follows form the first equality in (4.31) that

$$\begin{aligned} \begin{pmatrix} {{\widetilde{f}} }_j\\ g_j \end{pmatrix}\in I_{{{\widetilde{{\mathfrak {H}}}} }}-{z}_j{{\widetilde{V}} } \quad \Longleftrightarrow \quad \begin{pmatrix} {z}_j{{\widetilde{f}} }_j\\ {{\widetilde{f}} }_j-g_j \end{pmatrix}\in {{\widetilde{V}} }. \end{aligned}$$

Since ${{\widetilde{V}} }$ is a unitary relation in ${{\widetilde{{\mathfrak {H}}}} }$ one obtains ${{z}_j\overline{z}_k}[{{\widetilde{f}} }_j,{{\widetilde{f}} }_k]_{{{\widetilde{{\mathfrak {H}}}} }}=[{{\widetilde{f}} }_j-g_j,{{\widetilde{f}} }_k-g_k]_{{{\widetilde{{\mathfrak {H}}}} }}$ or, equivalently,

$$\begin{aligned} (1-{z}_j\overline{z}_k)[{{\widetilde{f}} }_j,{{\widetilde{f}} }_k]_{{{\widetilde{{\mathfrak {H}}}} }}=[{{\widetilde{f}} }_j,g_k]_{{{\widetilde{{\mathfrak {H}}}} }}+[g_j,{{\widetilde{f}} }_k]_{{{\widetilde{{\mathfrak {H}}}} }}-[g_j,g_k]_{{\mathfrak {H}}}. \end{aligned}$$

(4.32)

It follows from (4.30), (4.31), and (4.32) that

$$\begin{aligned} \begin{aligned}&\sum _{j,k=1}^{n}[\textsf {R}_{{z}_k}({z}_j)g_j,g_k]_{{\mathfrak {H}}}\xi _j\overline{\xi }_k=-\sum _{j,k=1}^n[{{\mathbf {K}}}_{{z}_j}g_j,{{\mathbf {K}}}_{{z}_k}g_k]_{{\mathfrak {H}}}\xi _j\overline{\xi }_k \\&\quad +\sum _{j,k=1}^n\frac{[{{\mathbf {K}}}_{{z}_j}g_j,g_k]_{{\mathfrak {H}}}+[g_j,{{\mathbf {K}}}_{{z}_k}g_k]_{{\mathfrak {H}}}-[g_j,g_k]_{{\mathfrak {H}}}}{1-{z}_j\overline{z}_k}{\xi _j\overline{\xi }_k}\\&\quad \overset{(4.31)}{=} \sum _{j,k=1}^n\left\{ \frac{[{{\widetilde{f}} }_j,g_k]_{{{\widetilde{{\mathfrak {H}}}} }}+[g_j,{{\widetilde{f}} }_k]_{{{\widetilde{{\mathfrak {H}}}} }}-[g_j,g_k]_{{\mathfrak {H}}}}{1-{z}_j\overline{z}_k} -[f_j,f_k]_{{\mathfrak {H}}}\right\} \xi _j\overline{\xi }_k\\&\quad \overset{(4.32)}{=}\sum _{j,k=1}^n\left\{ [{{\widetilde{f}} }_j,{{\widetilde{f}} }_k]_{{{\widetilde{{\mathfrak {H}}}} }}-[f_j,f_k]_{{{\mathfrak {H}}}}\right\} \xi _j\overline{\xi }_k\\&\quad =\sum _{j,k=1}^n\left[ (I-P_{{\mathfrak {H}}}){{\widetilde{f}} }_j,(I-P_{{\mathfrak {H}}}){{\widetilde{f}} }_k\right] _{{{\widetilde{{\mathfrak {H}}}} }}\xi _j\overline{\xi }_k. \end{aligned} \end{aligned}$$

(4.33)

This form has at most ${{\widetilde{\kappa }} }-\kappa$ negative squares, since $\mathrm{ind\,}({{\widetilde{{\mathfrak {H}}}} }[-]{{\mathfrak {H}}})={{\widetilde{\kappa }} }-\kappa$. Because the representation (4.29) is $({{\widetilde{\kappa }} }-\kappa )$-regular the set

$$\begin{aligned} \left\{ (I_{{{\widetilde{{\mathfrak {H}}}} }}-P_{{\mathfrak {H}}})(I_{{{\widetilde{{\mathfrak {H}}}} }}-{z}{{\widetilde{V}} })^{-1}{{\mathfrak {H}}}:{z}\in \rho ({{\widetilde{V}} }^{-1})\right\} \end{aligned}$$

is dense in ${{\widetilde{{\mathfrak {H}}}} }[-]{{\mathfrak {H}}}$ and hence it contains a $({{\widetilde{\kappa }} }-\kappa )$-dimensional negative subspace. Therefore, the form (4.33) has exactly ${{\widetilde{\kappa }} }-\kappa$ negative squares for an appropriate choice of ${z}_j$, $g_j$ $(j=1,2,\dots ,n)$. $\square$

Remark 4.7

In the case of a standard isometric operator the statement of Lemma 4.6 was proved in [28].

Theorem 4.8

Let $V:{{{\mathfrak {H}}}}\rightarrow {{{\mathfrak {H}}}}$ be an isometric operator, let $({{{\mathfrak {L}}}}_1\oplus {{{\mathfrak {L}}}}_2,\Gamma _1,\Gamma _2)$ be an ordinary boundary triple for V, and let $\Theta (\cdot )$, $\gamma _1(\cdot )$, $\gamma _2(\cdot )$ be the corresponding Weyl function and the $\gamma$-fields.

Then for $z\in \overline{{{\mathcal {D}}}}\cap \rho ({{\widetilde{V}} }^{-1})$ the formula

$$\begin{aligned} {\mathbf {K}}_z=(I_{{{\widetilde{{\mathfrak {H}}}} }}-z V_2)^{-1}+\gamma _2(1/z)\varepsilon (z)\left(I_{{{\mathfrak {L}}}_1}-\Theta ({\bar{z}})^*\varepsilon (z)\right)^{-1}\gamma _1({{\bar{z}}})^*, \end{aligned}$$

(4.34)

establishes a one-to-one correspondence between the set of ${{\widetilde{\kappa }} }-\kappa$-regular generalized coresolvents of V and the set of all operator-valued functions $\varepsilon (\cdot )\in {{\mathcal {S}}}_{{\widetilde{\kappa }}-\kappa }({{{\mathfrak {L}}}}_1,{{{\mathfrak {L}}}}_2)$, such that

$$\begin{aligned} 0\in \rho ({I_{{{\mathfrak {L}}}_1}-\Theta ({\bar{z}})^*\varepsilon (z)}). \end{aligned}$$

(4.35)

For $z\in \overline{{{\mathcal {D}}}}_e\cap \rho ({{\widetilde{V}} }^{-1})$ the formula (4.34) takes the form

$$\begin{aligned} {\mathbf {K}}_z=(I_{{{\widetilde{{\mathfrak {H}}}} }}-z V_1)^{-1}-\gamma _1(1/z) \varepsilon ^T(1/z)\left(I_{{{\mathfrak {L}}}_2}-\Theta (1/z)\varepsilon ^T(1/z)\right)^{-1}\gamma _2({{\bar{z}}})^*.\nonumber \\ \end{aligned}$$

(4.36)

Proof

The proof is divided into steps.

1. Verification that for every $\varepsilon (\cdot )\in {\mathcal {S}}_{{\widetilde{\kappa }}-\kappa }({{{\mathfrak {L}}}}_1,{{{\mathfrak {L}}}}_2)$ satisfying (4.35) the formula (4.34) determines a $({{\widetilde{\kappa }}}-\kappa )$-regular generalized coresolvent of V.

By Theorem 3.17 there exists a simple isometric operator $V^-$ in a Pontryagin space ${{\mathfrak {H}}}^-$ with negative index ${{\widetilde{\kappa }}}-\kappa$ and a unitary boundary pair $({{\mathfrak {L}}},\Gamma ^-)$ such that the corresponding Weyl function $\Theta ^-({z})$ coincides with $\varepsilon ({z})$ for ${z}\in {{\mathfrak {h}}}_\varepsilon =\rho (V_1^-)$, where $V_1^-={\mathrm{ker\,}\,}\Gamma _1^-$.

Next we construct a new unitary boundary pair $({{\widetilde{{{\mathfrak {L}}}}}},{{\widetilde{\Gamma }}})$ as the direct sum of the ordinary boundary triple $({{\mathfrak {L}}}, \Gamma ^+):=({{\mathfrak {L}}},\Gamma )$ and the unitary boundary pair $({{\mathfrak {L}}}, \Gamma ^-)$ by the formulas

$$\begin{aligned} {{\widetilde{{{\mathfrak {L}}}}}}= & {} {{\widetilde{{{\mathfrak {L}}}}}}_1\oplus {{\widetilde{{{\mathfrak {L}}}}}}_2, \text { where }{{\widetilde{{{\mathfrak {L}}}}}}_1={{\widetilde{{{\mathfrak {L}}}}}}_2=\begin{pmatrix} {{\mathfrak {L}}}_1\\ {{\mathfrak {L}}}_2 \end{pmatrix},\nonumber \\ {{\widetilde{\Gamma }}}= & {} \left\{ \left\{ \begin{pmatrix} f_+\\ f_-\\ f_+'\\ f'_- \end{pmatrix},\begin{pmatrix} u_1^+\\ u_2^-\\ u_1^-\\ u_2^+ \end{pmatrix}\right\} :\left\{ \begin{pmatrix} f_\pm \\ f_\pm ' \end{pmatrix},\begin{pmatrix} u_1^\pm \\ u_2^\pm \end{pmatrix}\right\} \in \Gamma ^\pm \right\} . \end{aligned}$$

(4.37)

Let $\varepsilon ({z})$, $\gamma _1^-({z})$, $\gamma _2^-({z})$ be the Weyl function and the $\gamma$-fields of the unitary boundary pair $({{\mathfrak {L}}},\Gamma ^-)$, let $\varepsilon ^T({z})$, $\gamma _1^{-,T}({z})$, $\gamma _2^{-,T}({z})$ be the Weyl function and the $\gamma$-fields of the transposed boundary pair $({{\mathfrak {L}}},(\Gamma ^-)^T)$ and let ${{\widetilde{V}} }_1={\mathrm{ker\,}\,}{{\widetilde{\Gamma }} }_1$, ${{\widetilde{V}} }_2={\mathrm{ker\,}\,}{{\widetilde{\Gamma }} }_2$. Then the Weyl function ${{\widetilde{\Theta }}}({z})$ and the $\gamma$-fields ${{\widetilde{\gamma }}}_1({z})$, ${{\widetilde{\gamma }}}_2({z})$ of the unitary boundary pair $({{\widetilde{{{\mathfrak {L}}}}}},{{\widetilde{\Gamma }}})$ are given by

$$\begin{aligned} {{\widetilde{\Theta }}}({z})= & {} \begin{pmatrix} 0&{}\varepsilon ^T({z})\\ \Theta ({z})&{}0 \end{pmatrix}: \begin{pmatrix} {{{\mathfrak {L}}}}_1\\ {{{\mathfrak {L}}}}_2 \end{pmatrix}\rightarrow \begin{pmatrix} {{{\mathfrak {L}}}}_1\\ {{{\mathfrak {L}}}}_2 \end{pmatrix},\quad {z}\in {{\widetilde{{{\mathcal {D}}}}}}:=\rho ({\widetilde{V}}_1)\cap {\mathbb {D}}, \end{aligned}$$

(4.38)

$$\begin{aligned} {{\widetilde{\gamma }}}_2({z}):= & {} \begin{pmatrix} 0&{}\gamma _2^+({z})\\ \gamma _2^{-,T}({z})&{}0 \end{pmatrix} :\begin{pmatrix} {{{\mathfrak {L}}}}_1\\ {{{\mathfrak {L}}}}_2 \end{pmatrix}\rightarrow \begin{pmatrix} {{\mathfrak {H}}}\\ {{{\mathfrak {H}}}}^- \end{pmatrix},\quad {z}\in {{\widetilde{{{\mathcal {D}}}}}}=\rho ({\widetilde{V}}_1)\cap {\mathbb {D}}, \end{aligned}$$

(4.39)

$$\begin{aligned} {{\widetilde{\gamma }}}_1({z})= & {} \begin{pmatrix} \gamma _1^+({z})&{}0\\ 0&{}\gamma _1^{-,T}({z}) \end{pmatrix} :\begin{pmatrix} {{{\mathfrak {L}}}}_1\\ {{{\mathfrak {L}}}}_2 \end{pmatrix}\rightarrow \begin{pmatrix} {{\mathfrak {H}}}\\ {{{\mathfrak {H}}}}^- \end{pmatrix},\quad {z}\in {{\widetilde{{{\mathcal {D}}}}}}_e:=\rho ({\widetilde{V}}_2)\cap {\mathbb {D}}_e.\nonumber \\ \end{aligned}$$

(4.40)

Consider the extension ${\widetilde{V}}_{\Phi }$ of the operator ${\widetilde{V}}=\begin{pmatrix} V&{}0\\ 0&{}V^- \end{pmatrix}$ in the space ${{\widetilde{{{\mathfrak {H}}}}}}:={{\mathfrak {H}}}^+\oplus {{\mathfrak {H}}}^-$, ${{\mathfrak {H}}}^+:={{\mathfrak {H}}}$, corresponding to the linear relation $\Phi$ of the form (4.7), where

$$\begin{aligned} \Phi _1=\Phi _2={I}_{{{\mathfrak {L}}}_1\oplus {{\mathfrak {L}}}_2}. \end{aligned}$$

In view of (4.37) the extension ${\widetilde{V}}_{\Phi }$ takes the form

$$\begin{aligned} {\widetilde{V}}_{\Phi }=\left\{ \left\{ \begin{pmatrix} f_+\\ f_-\\ \end{pmatrix},\begin{pmatrix} f_+'\\ f'_- \end{pmatrix}\right\} :\left\{ \begin{pmatrix} f_-\\ f_-' \end{pmatrix},\begin{pmatrix} \Gamma _2{{\widehat{f}} }_+\\ \Gamma _1{{\widehat{f}} }_+ \end{pmatrix}\right\} \in \Gamma ^-,\,{{\widehat{f}} }_+\in V^{-[*]}\right\} . \end{aligned}$$

Then $I_{{{\widetilde{{{\mathfrak {L}}}}}}_1}-{{\widetilde{\Theta }}}({\bar{z}})^*=\begin{pmatrix} I_{{{\mathfrak {L}}}_1}&{} -\Theta ({\bar{z}} )^*\\ -\varepsilon (z) &{} I_{{{\mathfrak {L}}}_2} \end{pmatrix}$ and the assumption (4.35) yields

$$\begin{aligned} 0\in \rho (I_{{{\widetilde{{{\mathfrak {L}}}}}}_1}-{{\widetilde{\Theta }}}({\bar{z}})^*) \end{aligned}$$

(4.41)

for ${{\bar{z}}}\in {{\widetilde{{{\mathcal {D}}}}}}$. By (4.41) and Corollary 4.3 one obtains $z\in \rho ({\widetilde{V}}_{\Phi }^{-1})$ and

$$\begin{aligned} (I_{{{\widetilde{{{\mathfrak {H}}}}}}}-{z} {\widetilde{V}}_{\Phi })^{-1}=(I_{{{\widetilde{{{\mathfrak {H}}}}}}}-{z} {\widetilde{V}}_2)^{-1}+{{\widetilde{\gamma }}}_2(1/z)(I_{{{\widetilde{{{\mathfrak {L}}}}}}_1}-{{\widetilde{\Theta }}}({{\bar{z}}})^*)^{-1}{{\widetilde{\gamma }}}_1({\bar{z}})^*. \end{aligned}$$

(4.42)

By using (4.40), (4.39), (4.42) and the formula (see [10][Prop. 2.8.7, p.108])

$$\begin{aligned} (I_{{{\widetilde{{{\mathfrak {L}}}}}}_1}-{{\widetilde{\Theta }}}({\bar{z}})^*)^{-1}= \begin{pmatrix} I_{{{\mathfrak {L}}}_1}&{} -\Theta ({\bar{z}} )^*\\ -\varepsilon (z) &{} I_{{{\mathfrak {L}}}_2} \end{pmatrix}^{-1}= \begin{pmatrix} * &{} * \\ \varepsilon ({z})(I_{{{\mathfrak {L}}}_1}-\Theta ({\bar{z}} )^*\varepsilon (z))^{-1}&{} * \end{pmatrix}\nonumber \\ \end{aligned}$$

(4.43)

one obtains the equality

$$\begin{aligned} \begin{aligned}&(I_{{{\widetilde{{{\mathfrak {H}}}}}}}-z {\widetilde{V}}_{\Phi })^{-1}=(I_{{{\widetilde{{{\mathfrak {H}}}}}}}-z{\widetilde{V}}_2)^{-1}\\&\quad + \begin{pmatrix} 0&{}\gamma _2(z)\\ \gamma ^{-,T}_2(z)&{}0 \end{pmatrix} \begin{pmatrix} * &{} * \\ \varepsilon ({z})(I_{{{\mathfrak {L}}}_1}-\Theta ({\bar{z}} )^*\varepsilon (z))^{-1} &{} * \end{pmatrix} \begin{pmatrix} \gamma _1({{\bar{z}}})^*&{}0\\ 0&{}\gamma ^{-,T}_1({{\bar{z}}})^* \end{pmatrix}, \end{aligned}\nonumber \\ \end{aligned}$$

(4.44)

where $*$ denotes blocks which are not used in further calculations. Considering the compression of the formula (4.44) to the subspace ${{\mathfrak {H}}}^+={{\mathfrak {H}}}$ one arrives at (4.34).

Similarly, for ${{\bar{z}}}\in {{\widetilde{{{\mathcal {D}}}}}}_e$ one obtains by (4.41) and Corollary 4.3 that $z\in \rho ({\widetilde{V}}_{\Phi }^{-1})$ and

$$\begin{aligned} (I_{{{\widetilde{{{\mathfrak {H}}}}}}}-z {\widetilde{V}}_{\Phi })^{-1}=(I_{{{\widetilde{{{\mathfrak {H}}}}}}}-{z} {\widetilde{V}}_1)^{-1}-{{\widetilde{\gamma }}}_1(1/{z})(I_{{{\widetilde{{{\mathfrak {L}}}}}}_1}-{{\widetilde{\Theta }}}(1/{z}))^{-1}{{\widetilde{\gamma }}}_2({\bar{z}})^*. \end{aligned}$$

By (4.43)

$$\begin{aligned} \left( I_{{{\widetilde{{{\mathfrak {L}}}}}}_1}-{{\widetilde{\Theta }}}\left( \frac{1}{z}\right) \right) ^{-1}= \begin{pmatrix} * &{} \omega _{12}(z)\\ * &{} * \end{pmatrix}, \quad \end{aligned}$$

(4.45)

where $\omega _{12}(z)=\varepsilon ^T(1/z)(I_{{{\mathfrak {L}}}_2}-\Theta (1/z)\varepsilon ^T(1/z))^{-1}$. Using (4.40), (4.39) and (4.45) one gets

$$\begin{aligned} \begin{aligned} (I_{{{\widetilde{{{\mathfrak {H}}}}}}}-&{z} {\widetilde{V}}_{\Phi })^{-1}=(I_{{{\widetilde{{{\mathfrak {H}}}}}}}-{z}{\widetilde{V}}_1)^{-1}\\&- \begin{pmatrix} \gamma _1(1/z)&{}0\\ 0&{}\gamma ^{-,T}_1(1/z) \end{pmatrix} \begin{pmatrix} * &{} \omega _{12}(z)\\ * &{} * \end{pmatrix} \begin{pmatrix} 0&{}\gamma ^{-,T}_2({{\bar{z}}})^*\\ \gamma _2({{\bar{z}}})^*&{}0 \end{pmatrix}. \end{aligned} \end{aligned}$$

(4.46)

The compression of the formula (4.46) to the subspace ${{\mathfrak {H}}}^+$ gives the equality (4.36).

2. Verification that every $({{\widetilde{\kappa }}}-\kappa )$- regular generalized coresolvent of V admits the representation (4.34), where $\varepsilon (\cdot )\in {{\mathcal {S}}}_{{{\widetilde{\kappa }}}-\kappa }({{{\mathfrak {L}}}}_1,{{{\mathfrak {L}}}}_2)$ and (4.35) holds.

First observe that for ${z}\in \rho ({\widetilde{V}}^{-1})$ and $g\in {{\mathfrak {H}}}$ the following relations hold:

$$\begin{aligned} \begin{pmatrix} {z}(I_{{{\widetilde{{\mathfrak {H}}}} }}-{z} {\widetilde{V}})^{-1} g\\ (I_{{{\widetilde{{\mathfrak {H}}}} }}-{z} {\widetilde{V}})^{-1}g-g \end{pmatrix}\in {\widetilde{V}},\quad \widehat{ {\mathbf {K}}}_{z} g:=\begin{pmatrix} {z}{\mathbf {K}}_{z} g\\ {\mathbf {K}}_{z} g-g \end{pmatrix}\in V^{-[*]}. \end{aligned}$$

(4.47)

Indeed, the first relation in (4.47) is self-evident and hence for every $h\in \mathrm{dom\,}V$ one also has the equality

$$\begin{aligned}{}[{z}(I_{{{\widetilde{{\mathfrak {H}}}} }}-{z} {\widetilde{V}})^{-1} g,h]_{{\mathfrak {H}}}=[ -g+(I_{{{\widetilde{{\mathfrak {H}}}} }}-{z} {\widetilde{V}})^{-1} g,Vh]_{{\mathfrak {H}}}. \end{aligned}$$

(4.48)

On the other hand,

$$\begin{aligned} \begin{aligned} {[{z}{{\mathbf {K}}}_{z} g,h]}_{{\mathfrak {H}}}&+[(I_{{{\mathfrak {H}}}}-{{\mathbf {K}}}_{z})g,V h]_{{\mathfrak {H}}}= [{{\mathbf {K}}}_{z} g,(\overline{z} I_{{{\mathfrak {H}}}}-V)h]_{{\mathfrak {H}}}+[g,Vh]_{{\mathfrak {H}}}\\&= \left[ (I_{{{\widetilde{{\mathfrak {H}}}} }}-{z} {\widetilde{V}})^{-1}g,(\overline{z} I_{{{\mathfrak {H}}}}-V)h\right] _{{\mathfrak {H}}}+[g,Vh]_{{\mathfrak {H}}}\\&=[{z}(I_{{{\widetilde{{\mathfrak {H}}}} }}-{z} {\widetilde{V}})^{-1}g,h]_{{\mathfrak {H}}}-[(I_{{{\widetilde{{\mathfrak {H}}}} }}-{z} {\widetilde{V}})^{-1}g,Vh]_{{\mathfrak {H}}}+[g,Vh]_{{\mathfrak {H}}}=0, \end{aligned} \end{aligned}$$

and here the last equality follows from (4.48). This proves the second relation in (4.47).

Next consider the linear relation

$$\begin{aligned} V_{z}:=\left\{ \widehat{ {\mathbf {K}}}_{z} g: \,g\in {{\mathfrak {H}}}\right\} ,\quad {z}\in \rho ({\widetilde{V}}^{-1}) \end{aligned}$$

(4.49)

in ${{\mathfrak {H}}}$. The linear relation $V_{z}$ is closed, since the assumptions

$$\begin{aligned} {z} {\mathbf {K}}_{z} g_n\rightarrow f,\quad {\mathbf {K}}_{z} g_n-g_n\rightarrow f'\quad (n\rightarrow \infty ) \end{aligned}$$

imply $g_n\rightarrow g:=\frac{1}{{z} }(f-{z} f')$ $({z} \ne 0)$ and hence

$$\begin{aligned} \begin{pmatrix} f\\ f' \end{pmatrix}= \begin{pmatrix} {z}{\mathbf {K}}_{z} g\\ {\mathbf {K}}_{z} g-g \end{pmatrix}\in V_{z}. \end{aligned}$$

In order to construct a parametric representation of the proper extension $V_{z}$ let us introduce a closed subspace ${{\mathcal {N}}}$ of $V_2$ such that

$$\begin{aligned} V_2=V\dotplus {{\mathcal {N}}}. \end{aligned}$$

Since $V^{-[*]}=V_1+V_2$, this implies

$$\begin{aligned} V^{-[*]}=V_1 \dotplus {{\mathcal {N}}}. \end{aligned}$$

(4.50)

Let $P_1$ and $P_2$ be projections onto the 1-st and the 2-nd components in ${{\mathfrak {H}}}^2$ and let $\Phi _j({z})$ be operator functions with values in B$({{\mathcal {N}}},{{\mathfrak {L}}}_j)$ defined by

$$\begin{aligned} \Phi _j({z})h=\Gamma _j\widehat{ {\mathbf {K}}}_{z} (P_1 h-{z} P_2h),\quad h\in {{\mathcal {N}}},\quad {z}\in \rho ({\widetilde{V}}^{-1}),\quad j=1,2. \end{aligned}$$

The values of $\Phi _j({z})$ belong to B$({{\mathcal {N}}},{{\mathfrak {L}}}_j)$ due to formula (4.47). Moreover, $\Phi _1({z})$ and $\Phi _2({z})$ satisfy (4.8), since the assumption $\Phi _1({z})h=\Phi _2({z})h=0$ implies $h\in V\cap {{\mathcal {N}}}=\{0\}$.

Now introduce the linear relation

$$\begin{aligned} \varepsilon ({z}):=\left\{ \left( {\begin{array}{c}\Phi _1({z})h\\ \Phi _2({z})h\end{array}}\right) :\,h\in {{\mathcal {N}}}\right\} \quad {z}\in \rho ({\widetilde{V}}^{-1}). \end{aligned}$$

(4.51)

Since $\mathrm{ran\,}(I_{{{\mathfrak {H}}}}-{z} V_2)={{\mathfrak {H}}}$ for all ${z}\in {{\mathcal {D}}}$ it follows from (4.49) and (4.51) that the linear relations $V_{z}$ and $\varepsilon ({z})$ are connected via (4.10) and hence

$$\begin{aligned} V_{z}=V_{\varepsilon ({z})}, \quad {z}\in \rho ({\widetilde{V}}^{-1}). \end{aligned}$$

(4.52)

Since $({{\mathfrak {L}}},\Gamma )$ is an ordinary boundary triple one concludes that $\varepsilon ({z})$ is closed in ${{\mathfrak {L}}}_1\oplus {{\mathfrak {L}}}_2$, and hence by Remark 4.4$\Phi _1({z})$ and $\Phi _2({z})$ satisfy (4.9); cf. [5, Theorem 2.1]. Using (4.47) and (4.52) one obtains

$$\begin{aligned} {\mathbf {K}}_{z} g=(I_{{{\mathfrak {H}}}}-{z} V_{\varepsilon ({z})})^{-1} g,\quad g\in {{\mathfrak {H}}}. \end{aligned}$$

(4.53)

Therefore ${z}\in \rho (V_{\varepsilon ({z})}^{-1})$ for all ${z}\in \rho ({\widetilde{V}}^{-1})$ and since $({{\mathfrak {L}}},\Gamma )$ is an ordinary boundary triple Corollary 4.3 shows that $0\in \rho (I_{{{\mathfrak {L}}}_1}-\Theta ({z})\varepsilon ({z}))$ and that the following formula holds for ${z}\in \rho ({\widetilde{V}}^{-1})\cap \overline{{{\mathcal {D}}}}$:

$$\begin{aligned} (I_{{{\mathfrak {H}}}}-{z} V_{\varepsilon ({z})})^{-1}=(I_{{{\mathfrak {H}}}}-{z} V_2)^{-1}+\gamma _2(1/{z})\Phi _2({z})(\Phi _1({z})-\Theta ({\bar{z}})^*\Phi _2({z}))^{-1}\gamma _1({\bar{z}})^*.\nonumber \\ \end{aligned}$$

(4.54)

It remains to show that $\varepsilon (\cdot )\in {{\mathcal {S}}}_{{{\widetilde{\kappa }}}-\kappa }({{{\mathfrak {L}}}}_1,{{{\mathfrak {L}}}}_2)$. For a choice of ${z}_j\in \rho ({\widetilde{V}}^{-1})\cap {{\mathbb {D}}}$ and $g_j\in {{\mathfrak {H}}}$ denote

$$\begin{aligned} {{\widehat{f}} }_{j}=\left( {\begin{array}{c}f_j\\ f_j'\end{array}}\right) :=\widehat{ {\mathbf {K}}}_{{z}_j} g_j,\quad j=1,\dots ,n, \end{aligned}$$

(4.55)

and let

$$\begin{aligned} \Gamma {{\widehat{f}} }_{j}=\left( {\begin{array}{c}\Phi _1({z}_j)h_j\\ \Phi _2({z}_j)h_j\end{array}}\right) ,\quad h_j\in {{\mathcal {N}}},\quad j=1,\dots ,n. \end{aligned}$$

(4.56)

Then it follows from (4.55), (4.56) and (3.2) that with $\xi _j\in {{\mathbb {C}}}$,

$$\begin{aligned} \sum _{j,k=1}^n a_{j,k}\xi _j\overline{\xi _k}&:=\sum _{j,k=1}^n\frac{(\Phi _1({z}_j)h_j,\Phi _1({z}_k)h_k)_{{{\mathfrak {L}}}_1}- (\Phi _2({z}_j)h_j,\Phi _2({z}_k)h_k)_{{{\mathfrak {L}}}_2}}{1-{z}_j\overline{{z}}_k} {\xi _j\overline{\xi _k}}\nonumber \\&=\sum _{j,k=1}^n\frac{(\Gamma _1{{\widehat{f}} }_{j},\Gamma _1{{\widehat{f}} }_{k})_{{{\mathfrak {L}}}_1}- (\Gamma _2{{\widehat{f}} }_{j},\Gamma _2{{\widehat{f}} }_{k})_{{{\mathfrak {L}}}_2}}{1-{z}_j\overline{{z}}_k} {\xi _j\overline{\xi _k}}\nonumber \\&=\sum _{j,k=1}^n\left\{ [ f_{j}, f_{k}]_{{{\mathfrak {H}}}}- [ f_{j}', f_{k}']_{{{\mathfrak {H}}}}\right\} \frac{\xi _j\overline{\xi _k}}{1-{z}_j\overline{{z}}_k}. \end{aligned}$$

(4.57)

Since

$$\begin{aligned} \begin{aligned}&{[} f_{j}, f_{k}]_{{{\mathfrak {H}}}}- [ f_{j}', f_{k}']_{{{\mathfrak {H}}}} ={{z}_j\overline{z}_k}[{{\mathbf {K}}}_{{z}_j}g_j,{{\mathbf {K}}}_{{z}_k}g_k]_{{\mathfrak {H}}}-[{{\mathbf {K}}}_{{z}_j}g_j-g_j,{{\mathbf {K}}}_{{z}_k}g_k-g_k]_{{\mathfrak {H}}}\\&\quad =({z}_j\overline{z}_k-1)[{{\mathbf {K}}}_{{z}_j}g_j,{{\mathbf {K}}}_{{z}_k}g_k]_{{\mathfrak {H}}}+[{{\mathbf {K}}}_{{z}_j}g_j,g_k]_{{\mathfrak {H}}}+[g_j,{{\mathbf {K}}}_{{z}_k}g_k]_{{\mathfrak {H}}}-[g_j,g_k]_{{\mathfrak {H}}}\\&\quad =({1-{z}_j\overline{z}_k})[\textsf {R}_{{z}_k}({z}_j)g_j,g_k]_{{\mathfrak {H}}}\end{aligned} \end{aligned}$$

the form in (4.57) is reduced to

$$\begin{aligned} \sum _{j,k=1}^n a_{j,k}\xi _j\overline{\xi _k}=\sum _{j,k=1}^n [\textsf {R}_{{z}_k}({z}_j)g_j,g_k]_{{\mathfrak {H}}}{\xi _j\overline{\xi _k}}. \end{aligned}$$

By Lemma 4.6 the form (4.57) has at most ${{\widetilde{\kappa }}}-\kappa$ and for some choice of ${z}_j$, $g_j$, $j=1,\dots ,n$, exactly ${{\widetilde{\kappa }}}-\kappa$ negative squares.

In view of Lemma 3.2 in [16] this implies that the operator $\Phi _1({z})$ is invertible for all ${z}\in \rho ({\widetilde{V}}^{-1})\cap {{\mathbb {D}}}$ except ${{\widetilde{\kappa }}}-\kappa$ points and

$$\begin{aligned} \varepsilon ({z})=\Phi _2({z})\Phi _1({z})^{-1}\in {{\mathcal {S}}}_{\widetilde{\kappa }-\kappa }({{\mathfrak {L}}}_1,{{\mathfrak {L}}}_2). \end{aligned}$$

(4.58)

In view of (4.58) the formula (4.54) can be rewritten as (4.34). $\square$

Remark 4.9

• (1) For a standard isometric operator in a Pontryagin (resp. Kreĭn) space similar formulas for generalized coresolvents were found in [38,39,40] (resp. [28]). For the case of a nonstandard isometric operator in a Pontryagin space see  [44]. An elegant proof of the formula for generalized resolvents of a nonstandard Pontryagin space symmetric operator with deficiency index (1,1) given by H. de Snoo was presented in [34]. In [5] a description of regular generalized resolvents of a nonstandard Pontryagin space isometric operator was given by the method of boundary triples. For a Hilbert space isometric operator this method was developed earlier in [41] and applied to the proof of Kreĭn type resolvent formulas (4.34), (4.36).

• (2) The extension $V_{\varepsilon ({z})}$ appearing in (4.52) is an analog of Shtraus extension, which was introduced in [47] for the case of a symmetric operator. In view of (4.53) the vector function $f_{z}=\mathbf{K}_{z} g$ can be treated as a solution of the following ”abstract boundary value problem” with z-dependent boundary conditions

  $$\begin{aligned} {{\widehat{f}} }_{z}:=\left( {\begin{array}{c}{z} f_{z}\\ f_{z}-g\end{array}}\right) \in V^{-[*]},\quad \Gamma _2 {{\widehat{f}} }_{z}=\varepsilon ({z})\Gamma _1 {{\widehat{f}} }_{z}. \end{aligned}$$

• (3) In abstract interpolation problem considered in [36] the crucial role was played by the Arov-Grossman formula for scattering matrices of unitary extensions of isometric operators, [3]. In [6] the formula for generalized coresolvents was applied to the description of scattering matrices of unitary extensions of Pontryagin space isometric operators which, in turn, was used in [7] for parametrization of solutions of an indefinite abstract interpolation problem, see also [15]. The present version of formula (4.34) will allow to consider k-regular indefinite interpolation problems with the growth of index $\kappa$.